2024-03-25T16:02:12.888 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:02:12.888 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238 --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/orig.config.yaml 2024-03-25T16:02:12.931 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:02:13.014 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621238' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK5AQGusdjqA96uup2BCfl1loIxJfl6NhrqvGoVH2U8OcUsKi/skdMWIWIDHuAdniI9UIgaBxVWJZrsc7DcoN4s= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcLfbwlvh5E0m3Gu/sf3hW7GNawTRIu453B3YyRAHMbkxUusdsYvD1JLkJPyoQpP75IrNRy6bdn/EADUKCRZLQ= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:02:13.015 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:02:13.015 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:02:13.015 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:02:13.017 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:02:13.044 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:02:13.044 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:02:13.044 INFO:teuthology.packaging:ref: None 2024-03-25T16:02:13.044 INFO:teuthology.packaging:tag: None 2024-03-25T16:02:13.044 INFO:teuthology.packaging:branch: main 2024-03-25T16:02:13.044 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:02:13.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:02:13.418 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:02:13.420 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:02:13.427 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:02:13.427 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:02:13.462 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:02:13.481 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:02:13.487 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:02:13.514 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:56:37.599907', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK5AQGusdjqA96uup2BCfl1loIxJfl6NhrqvGoVH2U8OcUsKi/skdMWIWIDHuAdniI9UIgaBxVWJZrsc7DcoN4s='} 2024-03-25T16:02:13.538 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:56:37.597959', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcLfbwlvh5E0m3Gu/sf3hW7GNawTRIu453B3YyRAHMbkxUusdsYvD1JLkJPyoQpP75IrNRy6bdn/EADUKCRZLQ='} 2024-03-25T16:02:13.538 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:02:13.544 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:02:13.545 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:02:13.545 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:02:13.638 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3c4a2940d0>, signals=[15]) 2024-03-25T16:02:13.639 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:02:13.644 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:02:13.645 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T16:02:13.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:02:13.762 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T16:02:13.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:02:13.851 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:02:13.858 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-03-25T16:02:13.876 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-03-25T16:02:13.876 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-03-25T16:02:13.934 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:02:13.935 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:02:13.936 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-03-25T16:02:13.960 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-03-25T16:02:13.978 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-03-25T16:02:13.979 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-03-25T16:02:14.037 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-03-25T16:02:14.037 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-03-25T16:02:14.037 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-03-25T16:02:14.038 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:02:14.038 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-03-25T16:02:14.038 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:02:14.038 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:02:14.038 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-03-25T16:02:14.039 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:02:14.039 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:02:14.039 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:02:14.039 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:02:14.039 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:02:14.039 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:02:14.040 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-03-25T16:02:14.077 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:02:14.087 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:02:14.093 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:02:14.093 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:02:14.095 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:02:14.112 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:02:14.118 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:02:14.119 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:02:14.152 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:02:14.263 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:02:14.316 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:02:14.330 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:02:14.330 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:02:14.330 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:02:14.331 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:02:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:02:14.348 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-03-25T16:02:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:02:14.349 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-03-25T16:02:14.405 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:02:14.405 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.14.0-361.el9.x86_64 2024-03-25T16:02:14.405 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2024-03-25T16:02:14.407 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:02:14.407 INFO:teuthology.task.kernel:Running kernel on smithi044: 5.14.0-361.el9.x86_64 2024-03-25T16:02:14.407 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum install -y kernel 2024-03-25T16:02:16.654 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T16:02:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:02:18.451 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-25T16:02:18.471 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-25T16:02:19.692 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-03-25T16:02:20.171 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 8.0 MB 00:01 2024-03-25T16:02:22.084 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-25T16:02:22.632 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-25T16:02:26.702 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.1 MB 00:01 2024-03-25T16:02:27.672 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-03-25T16:02:28.161 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-25T16:02:29.498 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-25T16:02:29.579 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T16:02:33.024 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 21 MB 00:03 2024-03-25T16:02:34.588 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:02:36.685 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:02:36.686 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:02:36.775 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:02:36.785 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:02:36.786 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:02:36.786 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:02:36.786 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:02:36.786 INFO:teuthology.orchestra.run.smithi111.stdout:Install 4 Packages 2024-03-25T16:02:36.786 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:02:36.786 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 76 M 2024-03-25T16:02:36.786 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 123 M 2024-03-25T16:02:36.787 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:02:36.922 INFO:teuthology.orchestra.run.smithi111.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 771 kB/s | 44 kB 00:00 2024-03-25T16:02:37.855 INFO:teuthology.orchestra.run.smithi111.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 15 MB/s | 15 MB 00:00 2024-03-25T16:02:38.106 INFO:teuthology.orchestra.run.smithi111.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 23 MB/s | 27 MB 00:01 2024-03-25T16:02:38.166 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:02:38.406 INFO:teuthology.orchestra.run.smithi111.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 22 MB/s | 33 MB 00:01 2024-03-25T16:02:38.406 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:02:38.406 INFO:teuthology.orchestra.run.smithi111.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-25T16:02:39.066 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:02:39.436 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:02:39.436 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:02:40.365 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:02:40.365 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:02:40.426 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:02:40.426 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:02:40.455 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:02:40.464 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:02:40.465 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:02:40.465 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:02:40.465 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:02:40.465 INFO:teuthology.orchestra.run.smithi044.stdout:Install 4 Packages 2024-03-25T16:02:40.465 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:02:40.465 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 76 M 2024-03-25T16:02:40.465 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 123 M 2024-03-25T16:02:40.465 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:02:40.834 INFO:teuthology.orchestra.run.smithi044.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 225 kB/s | 44 kB 00:00 2024-03-25T16:02:42.401 INFO:teuthology.orchestra.run.smithi044.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 8.4 MB/s | 15 MB 00:01 2024-03-25T16:02:42.949 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:02:43.201 INFO:teuthology.orchestra.run.smithi044.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 13 MB/s | 33 MB 00:02 2024-03-25T16:02:43.558 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:02:43.648 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:02:43.693 INFO:teuthology.orchestra.run.smithi044.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 9.6 MB/s | 27 MB 00:02 2024-03-25T16:02:43.693 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:02:43.693 INFO:teuthology.orchestra.run.smithi044.stdout:Total 23 MB/s | 76 MB 00:03 2024-03-25T16:02:44.387 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:02:44.788 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:02:44.788 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:02:45.827 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:02:45.827 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:02:46.441 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:02:46.617 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:02:48.444 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:02:49.072 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:02:49.160 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:02:51.322 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:02:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:02:52.043 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:02:52.181 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:02:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:02:57.246 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:02:57.289 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:02.395 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:25.889 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:25.889 INFO:teuthology.orchestra.run.smithi111.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:03:25.889 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:03:25.933 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:26.265 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:26.265 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:03:26.265 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:26.266 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:03:26.713 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:03:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:03:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:03:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:03:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:03:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:03:26.714 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:03:27.063 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:03:27.183 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:03:27.184 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:03:27.184 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:03:27.184 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:03:27.184 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:03:27.184 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:03:27.184 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:03:27.184 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:03:27.263 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:03:27.263 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:03:27.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2024-03-25T16:03:27.308 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2024-03-25T16:03:27.308 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2024-03-25T16:03:27.308 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.06 2024-03-25T16:03:27.308 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 68.el9 2024-03-25T16:03:27.308 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2024-03-25T16:03:27.308 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:03:27.308 INFO:teuthology.orchestra.run.smithi111.stdout:Group : Unspecified 2024-03-25T16:03:27.308 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 8264254 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : builder@centos.org 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:03:27.309 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2024-03-25T16:03:27.310 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:03:27.310 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:03:27.310 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:03:27.311 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:03:27.311 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:03:27.898 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2024-03-25T16:03:29.295 INFO:teuthology.orchestra.run.smithi111.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:03:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:done 2024-03-25T16:03:29.338 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-03-25T16:03:29.352 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.tb1AViZP1V 2024-03-25T16:03:29.353 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tb1AViZP1V 2024-03-25T16:03:29.421 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.tb1AViZP1V 2024-03-25T16:03:29.564 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.tb1AViZP1V is 6KB 2024-03-25T16:03:29.614 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.tb1AViZP1V 2024-03-25T16:03:29.630 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:03:29.694 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:03:29.695 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:03:29.695 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:03:29.695 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:03:29.696 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:03:29.842 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2024-03-25T16:03:33.159 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:33.160 INFO:teuthology.orchestra.run.smithi044.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:03:33.160 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:03:33.205 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:33.733 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:33.733 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:03:33.733 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:33.733 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:34.135 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:34.135 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:03:34.135 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:03:34.136 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:03:34.136 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:03:34.136 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:03:34.136 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:03:34.136 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:03:34.136 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:03:34.477 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:03:34.751 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:03:34.751 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:03:34.751 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:03:34.751 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:03:34.751 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:03:34.751 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:03:34.751 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:03:34.751 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:03:34.834 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:03:34.834 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:03:34.835 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2024-03-25T16:03:34.872 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.06 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 68.el9 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Group : Unspecified 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 8264254 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : builder@centos.org 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2024-03-25T16:03:34.873 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:03:34.874 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:03:34.875 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:03:34.875 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:03:34.875 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:03:35.490 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2024-03-25T16:03:37.036 INFO:teuthology.orchestra.run.smithi044.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:03:37.058 INFO:teuthology.orchestra.run.smithi044.stderr:done 2024-03-25T16:03:37.060 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-03-25T16:03:37.075 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.NnB5XicIpB 2024-03-25T16:03:37.075 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NnB5XicIpB 2024-03-25T16:03:37.148 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.NnB5XicIpB 2024-03-25T16:03:37.294 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.NnB5XicIpB is 6KB 2024-03-25T16:03:37.341 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.NnB5XicIpB 2024-03-25T16:03:37.356 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:03:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:03:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:03:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:03:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:03:37.428 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:03:37.579 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2024-03-25T16:03:59.866 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:03:59.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T16:03:59.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:04:07.586 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:04:07.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T16:04:07.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:04:34.463 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-25T16:04:43.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T16:04:43.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:04:46.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-25T16:04:58.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T16:04:58.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:05:07.598 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:05:16.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T16:05:16.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:05:16.871 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-03-25T16:05:17.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T16:05:17.513 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:05:17.513 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-03-25T16:05:17.527 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:05:17.527 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:05:17.527 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:05:17.527 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:05:18.529 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:05:18.529 DEBUG:teuthology.orchestra.run.smithi044:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:05:18.629 INFO:teuthology.orchestra.run.smithi044.stdout:ttyS1 2024-03-25T16:05:18.675 DEBUG:teuthology.parallel:result is None 2024-03-25T16:05:30.325 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-03-25T16:05:30.863 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T16:05:30.863 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:05:30.863 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-03-25T16:05:30.890 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:05:30.890 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:05:30.890 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:05:30.890 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:05:31.891 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:05:31.891 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:05:31.950 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-03-25T16:05:31.982 DEBUG:teuthology.parallel:result is None 2024-03-25T16:05:31.983 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:05:31.990 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:05:31.990 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:05:31.994 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:05:32.010 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:05:32.072 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:05:32.079 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:05:32.079 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:05:32.081 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:05:32.142 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:05:32.149 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:05:32.149 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:05:32.151 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:05:32.183 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:05:32.197 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:05:32.199 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:05:32.212 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:05:32.213 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:05:32.221 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:05:32.221 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:05:32.242 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:05:32.269 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:05:32.278 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:05:32.278 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:05:32.312 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:05:32.327 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:05:32.414 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:05:32.459 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:05:32.542 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:05:32.584 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T16:05:32.584 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:05:32.652 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:05:32.694 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:05:32.742 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:05:32.827 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:05:32.872 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T16:05:32.872 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:05:32.942 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-03-25T16:05:32.944 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-03-25T16:05:33.002 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:05:33.038 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:05:33.392 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:05:33.402 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:05:33.402 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:05:33.414 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:05:33.423 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-03-25T16:05:33.485 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-03-25T16:05:33.487 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-03-25T16:05:33.535 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-03-25T16:05:33.537 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:05:33.537 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-03-25T16:05:33.562 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-03-25T16:05:33.562 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-03-25T16:05:33.607 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-03-25T16:05:33.608 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:05:33.608 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:05:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:05:33.638 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:05:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:05:33.678 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:05:33.679 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:05:33.714 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:05:33.751 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:05:33.761 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:05:33.779 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:05:33.781 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi111.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:12:03.932 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com')] 2024-03-25T16:12:03.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T16:12:03.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:04.041 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-03-25T16:12:04.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T16:12:04.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T16:12:04.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:04.216 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-03-25T16:12:04.293 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T16:12:04.293 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:12:04.304 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:12:04.304 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:12:04.304 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:12:04.306 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:12:04.306 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:12:04.337 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:12:04.354 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:12:04.365 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:12:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:12:04.383 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-03-25T16:12:04.397 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-03-25T16:12:04.413 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-03-25T16:12:04.414 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:12:04.426 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-03-25T16:12:04.431 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:12:04.443 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:12:04.459 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:12:04.488 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:12:04.491 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:12:04.491 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-03-25T16:12:04.491 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:12:04.491 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:12:04.491 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:12:04.491 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:12:04.512 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:12:04.515 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:12:04.515 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-03-25T16:12:04.515 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:12:04.515 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:12:04.515 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:12:04.515 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:12:04.516 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:12:04.525 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:12:04.525 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:12:04.525 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:12:04.525 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:12:04.530 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:12:04.531 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:12:04.531 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:12:04.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:12:04.533 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:12:04.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:12:04.810 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:12:04.810 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:12:04.818 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:12:04.818 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:12:04.927 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:12:04.927 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T16:12:04.927 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:12:04.963 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:12:04.964 DEBUG:teuthology.orchestra.run.smithi044:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:12:05.038 DEBUG:teuthology.orchestra.run.smithi044:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:12:05.059 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:12:05.060 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T16:12:05.060 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:12:05.091 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:12:05.091 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:12:05.124 DEBUG:teuthology.orchestra.run.smithi044:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:12:05.167 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:12:05.199 INFO:teuthology.orchestra.run.smithi044.stdout:check_obsoletes = 1 2024-03-25T16:12:05.201 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum clean all 2024-03-25T16:12:05.250 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:12:05.322 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-03-25T16:12:05.324 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-03-25T16:12:05.444 INFO:teuthology.orchestra.run.smithi044.stdout:74 files removed 2024-03-25T16:12:05.471 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw 2024-03-25T16:12:05.575 INFO:teuthology.orchestra.run.smithi111.stdout:74 files removed 2024-03-25T16:12:05.602 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw 2024-03-25T16:12:06.298 INFO:teuthology.orchestra.run.smithi044.stdout:ceph packages for x86_64 157 kB/s | 82 kB 00:00 2024-03-25T16:12:06.442 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 151 kB/s | 82 kB 00:00 2024-03-25T16:12:06.782 INFO:teuthology.orchestra.run.smithi044.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-25T16:12:06.835 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-25T16:12:07.166 INFO:teuthology.orchestra.run.smithi044.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-25T16:12:07.253 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-25T16:12:08.246 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - AppStream 7.0 MB/s | 6.7 MB 00:00 2024-03-25T16:12:08.260 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - AppStream 6.3 MB/s | 6.7 MB 00:01 2024-03-25T16:12:09.655 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-25T16:12:09.736 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-25T16:12:11.329 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.0 MB 00:01 2024-03-25T16:12:12.468 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-03-25T16:12:12.989 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-25T16:12:16.636 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 19 MB 00:03 2024-03-25T16:12:16.922 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:12:18.382 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-25T16:12:20.094 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T16:12:21.086 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.1 MB 00:00 2024-03-25T16:12:22.495 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-03-25T16:12:23.272 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-03-25T16:12:25.142 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T16:12:27.572 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:12:27.585 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:12:27.585 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:12:27.586 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:12:27.587 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-03-25T16:12:27.588 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:12:27.589 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:12:27.589 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:12:27.589 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:12:27.589 INFO:teuthology.orchestra.run.smithi111.stdout:Install 30 Packages 2024-03-25T16:12:27.589 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-03-25T16:12:27.589 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:12:27.589 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 58 M 2024-03-25T16:12:27.589 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:12:28.381 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T16:12:30.905 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:12:30.921 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:12:30.922 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:12:30.923 INFO:teuthology.orchestra.run.smithi044.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout:Installing weak dependencies: 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout:Install 30 Packages 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 2 Packages 2024-03-25T16:12:30.924 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:12:30.925 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 58 M 2024-03-25T16:12:30.925 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:12:32.176 INFO:teuthology.orchestra.run.smithi111.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 5.3 MB 00:04 2024-03-25T16:12:32.301 INFO:teuthology.orchestra.run.smithi111.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 202 kB/s | 25 kB 00:00 2024-03-25T16:12:32.718 INFO:teuthology.orchestra.run.smithi111.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.7 MB/s | 742 kB 00:00 2024-03-25T16:12:33.018 INFO:teuthology.orchestra.run.smithi111.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.6 MB/s | 498 kB 00:00 2024-03-25T16:12:33.399 INFO:teuthology.orchestra.run.smithi044.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.4 MB/s | 5.3 MB 00:02 2024-03-25T16:12:33.482 INFO:teuthology.orchestra.run.smithi044.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 303 kB/s | 25 kB 00:00 2024-03-25T16:12:33.716 INFO:teuthology.orchestra.run.smithi044.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.1 MB/s | 742 kB 00:00 2024-03-25T16:12:33.858 INFO:teuthology.orchestra.run.smithi044.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.4 MB/s | 498 kB 00:00 2024-03-25T16:12:33.885 INFO:teuthology.orchestra.run.smithi111.stdout:(5/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 3.3 MB/s | 20 MB 00:05 2024-03-25T16:12:33.960 INFO:teuthology.orchestra.run.smithi111.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 604 kB/s | 45 kB 00:00 2024-03-25T16:12:34.044 INFO:teuthology.orchestra.run.smithi111.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-25T16:12:34.127 INFO:teuthology.orchestra.run.smithi111.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.9 MB/s | 162 kB 00:00 2024-03-25T16:12:34.294 INFO:teuthology.orchestra.run.smithi111.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.9 MB/s | 320 kB 00:00 2024-03-25T16:12:34.453 INFO:teuthology.orchestra.run.smithi111.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.8 MB/s | 298 kB 00:00 2024-03-25T16:12:34.545 INFO:teuthology.orchestra.run.smithi111.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-25T16:12:34.645 INFO:teuthology.orchestra.run.smithi111.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-25T16:12:34.695 INFO:teuthology.orchestra.run.smithi111.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:12:34.729 INFO:teuthology.orchestra.run.smithi111.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:12:34.796 INFO:teuthology.orchestra.run.smithi111.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:12:34.838 INFO:teuthology.orchestra.run.smithi111.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:12:34.871 INFO:teuthology.orchestra.run.smithi111.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:12:34.947 INFO:teuthology.orchestra.run.smithi111.stdout:(18/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 5.1 MB 00:01 2024-03-25T16:12:35.030 INFO:teuthology.orchestra.run.smithi111.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 419 kB/s | 35 kB 00:00 2024-03-25T16:12:35.058 INFO:teuthology.orchestra.run.smithi111.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-25T16:12:35.066 INFO:teuthology.orchestra.run.smithi044.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 4.2 MB/s | 5.1 MB 00:01 2024-03-25T16:12:35.083 INFO:teuthology.orchestra.run.smithi111.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 850 kB/s | 44 kB 00:00 2024-03-25T16:12:35.142 INFO:teuthology.orchestra.run.smithi044.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 602 kB/s | 45 kB 00:00 2024-03-25T16:12:35.217 INFO:teuthology.orchestra.run.smithi044.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-25T16:12:35.292 INFO:teuthology.orchestra.run.smithi044.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-25T16:12:35.376 INFO:teuthology.orchestra.run.smithi044.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 320 kB 00:00 2024-03-25T16:12:35.458 INFO:teuthology.orchestra.run.smithi111.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 772 kB/s | 309 kB 00:00 2024-03-25T16:12:35.459 INFO:teuthology.orchestra.run.smithi044.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 298 kB 00:00 2024-03-25T16:12:35.509 INFO:teuthology.orchestra.run.smithi111.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-03-25T16:12:35.535 INFO:teuthology.orchestra.run.smithi044.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-25T16:12:35.568 INFO:teuthology.orchestra.run.smithi111.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-03-25T16:12:35.617 INFO:teuthology.orchestra.run.smithi111.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-25T16:12:35.710 INFO:teuthology.orchestra.run.smithi044.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 605 kB/s | 106 kB 00:00 2024-03-25T16:12:35.810 INFO:teuthology.orchestra.run.smithi044.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-25T16:12:35.877 INFO:teuthology.orchestra.run.smithi044.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 662 kB/s | 44 kB 00:00 2024-03-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi111.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 838 kB 00:00 2024-03-25T16:12:36.102 INFO:teuthology.orchestra.run.smithi044.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 661 kB 00:00 2024-03-25T16:12:36.186 INFO:teuthology.orchestra.run.smithi044.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-25T16:12:36.210 INFO:teuthology.orchestra.run.smithi111.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-25T16:12:36.228 INFO:teuthology.orchestra.run.smithi044.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 993 kB/s | 41 kB 00:00 2024-03-25T16:12:36.420 INFO:teuthology.orchestra.run.smithi044.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-25T16:12:36.462 INFO:teuthology.orchestra.run.smithi044.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 841 kB/s | 35 kB 00:00 2024-03-25T16:12:36.504 INFO:teuthology.orchestra.run.smithi044.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:12:36.618 INFO:teuthology.orchestra.run.smithi111.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-03-25T16:12:36.763 INFO:teuthology.orchestra.run.smithi044.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-25T16:12:36.944 INFO:teuthology.orchestra.run.smithi111.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-03-25T16:12:36.989 INFO:teuthology.orchestra.run.smithi044.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-25T16:12:37.032 INFO:teuthology.orchestra.run.smithi044.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 622 kB/s | 26 kB 00:00 2024-03-25T16:12:37.082 INFO:teuthology.orchestra.run.smithi044.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 977 kB/s | 49 kB 00:00 2024-03-25T16:12:37.131 INFO:teuthology.orchestra.run.smithi044.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-25T16:12:37.198 INFO:teuthology.orchestra.run.smithi044.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T16:12:37.257 INFO:teuthology.orchestra.run.smithi044.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T16:12:37.340 INFO:teuthology.orchestra.run.smithi044.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-25T16:12:38.085 INFO:teuthology.orchestra.run.smithi111.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.3 MB/s | 3.4 MB 00:01 2024-03-25T16:12:38.135 INFO:teuthology.orchestra.run.smithi111.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 3.1 MB 00:01 2024-03-25T16:12:38.298 INFO:teuthology.orchestra.run.smithi044.stdout:(29/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.4 MB/s | 9.7 MB 00:07 2024-03-25T16:12:38.574 INFO:teuthology.orchestra.run.smithi044.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.7 MB/s | 3.4 MB 00:01 2024-03-25T16:12:38.936 INFO:teuthology.orchestra.run.smithi111.stdout:(32/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 903 kB/s | 9.7 MB 00:11 2024-03-25T16:12:38.941 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:38.941 INFO:teuthology.orchestra.run.smithi111.stdout:Total 5.1 MB/s | 58 MB 00:11 2024-03-25T16:12:38.977 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:12:39.015 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:12:39.015 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:12:39.491 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:12:39.492 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:12:39.507 INFO:teuthology.orchestra.run.smithi044.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 3.1 MB 00:01 2024-03-25T16:12:40.941 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:12:41.083 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:12:41.132 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:12:41.158 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:12:41.188 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:41.206 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:41.349 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:41.411 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:41.734 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:12:41.784 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:41.916 INFO:teuthology.orchestra.run.smithi044.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 20 MB 00:10 2024-03-25T16:12:41.920 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:41.920 INFO:teuthology.orchestra.run.smithi044.stdout:Total 5.3 MB/s | 58 MB 00:10 2024-03-25T16:12:41.949 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:12:41.955 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:12:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:41.996 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:12:41.996 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:12:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:42.061 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:12:42.114 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:12:42.199 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:42.221 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:42.284 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:42.311 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:12:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:12:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:12:42.486 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:12:42.487 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:12:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:12:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:12:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:12:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:12:42.906 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:12:42.936 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:43.051 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:43.082 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:12:43.121 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:12:43.570 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:12:43.597 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:43.668 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:43.744 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:12:43.782 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:12:43.994 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:12:44.094 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:12:44.129 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:12:44.154 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:12:44.177 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:44.193 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:44.385 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:44.442 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:44.787 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:12:44.839 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:44.932 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:45.022 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:12:45.049 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:45.112 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:45.141 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:12:45.185 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:12:45.255 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:45.276 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:45.340 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:45.366 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:12:45.478 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:12:45.534 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:12:45.575 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:12:45.608 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:12:45.637 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:12:45.669 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:12:45.926 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:45.963 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:45.995 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:12:46.024 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:46.098 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:46.129 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:12:46.175 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:46.607 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:46.653 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:12:46.681 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:46.681 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:12:46.682 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:12:46.682 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:12:46.685 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:46.717 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:46.756 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:46.826 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:12:46.862 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:12:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:49.112 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:49.687 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:49.721 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:49.813 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:49.814 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:12:49.814 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:12:49.814 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:12:49.847 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:56.064 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-03-25T16:12:56.064 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-03-25T16:12:56.064 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-03-25T16:12:56.064 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-03-25T16:12:56.064 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-03-25T16:12:56.064 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /root 2024-03-25T16:12:56.064 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-03-25T16:12:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:12:56.941 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:12:56.966 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:12:56.966 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:56.966 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:12:56.967 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:12:56.967 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:12:56.967 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:12:56.969 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:12:57.052 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:12:57.052 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:12:58.187 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:12:58.188 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:12:58.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:12:58.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:12:58.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:12:58.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:12:58.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:12:58.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:58.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:12:58.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.211 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.212 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:12:59.213 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:12:59.460 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test 2024-03-25T16:12:59.604 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:59.605 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /sys 2024-03-25T16:12:59.605 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /proc 2024-03-25T16:12:59.605 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /mnt 2024-03-25T16:12:59.605 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /var/tmp 2024-03-25T16:12:59.605 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /home 2024-03-25T16:12:59.605 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /root 2024-03-25T16:12:59.605 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /tmp 2024-03-25T16:12:59.605 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:00.010 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:35 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:00.165 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:13:00.166 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:13:00.167 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:00.167 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:13:00.167 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:00.167 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-03-25T16:13:00.167 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:00.167 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 48 M 2024-03-25T16:13:00.167 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 204 M 2024-03-25T16:13:00.167 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:13:00.485 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T16:13:00.534 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:13:00.558 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:13:00.558 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:00.558 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:13:00.558 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:13:00.558 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:13:00.559 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:00.560 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:13:00.577 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-03-25T16:13:00.619 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T16:13:00.653 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:13:00.653 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:13:00.778 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T16:13:00.894 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 163 kB 00:00 2024-03-25T16:13:01.794 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:13:01.794 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:13:01.794 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:13:01.795 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:13:01.796 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:13:01.797 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:13:01.797 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:13:01.797 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:13:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:13:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:02.733 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-03-25T16:13:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.733 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:13:02.733 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.734 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:13:02.735 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:13:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:13:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:13:02.736 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:02.736 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:03.039 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-test 2024-03-25T16:13:03.610 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:00:35 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:03.774 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:03.774 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:03.775 INFO:teuthology.orchestra.run.smithi044.stdout:Install 6 Packages 2024-03-25T16:13:03.776 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:03.776 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 48 M 2024-03-25T16:13:03.776 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 204 M 2024-03-25T16:13:03.776 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:13:04.246 INFO:teuthology.orchestra.run.smithi044.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-25T16:13:04.305 INFO:teuthology.orchestra.run.smithi044.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-25T16:13:04.330 INFO:teuthology.orchestra.run.smithi044.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T16:13:04.563 INFO:teuthology.orchestra.run.smithi044.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 321 kB/s | 163 kB 00:00 2024-03-25T16:13:04.672 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 47 MB 00:04 2024-03-25T16:13:04.674 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:04.675 INFO:teuthology.orchestra.run.smithi111.stdout:Total 11 MB/s | 48 MB 00:04 2024-03-25T16:13:04.689 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:13:04.709 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:13:04.710 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:13:04.839 INFO:teuthology.orchestra.run.smithi044.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 368 kB/s | 187 kB 00:00 2024-03-25T16:13:05.092 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:13:05.093 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:13:05.895 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:13:05.914 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:13:06.889 INFO:teuthology.orchestra.run.smithi044.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-25T16:13:06.892 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:06.892 INFO:teuthology.orchestra.run.smithi044.stdout:Total 15 MB/s | 48 MB 00:03 2024-03-25T16:13:06.906 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:13:06.929 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:13:06.930 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:13:07.325 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:13:07.326 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:13:07.916 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:13:07.982 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:13:08.017 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:13:08.049 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:13:08.090 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:13:08.117 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:13:10.398 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:13:10.481 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:13:12.844 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:13:12.927 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:13:12.967 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:13:12.967 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:13:12.967 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:13:12.967 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:13:12.967 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:13:12.968 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:13.343 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:13.694 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph 2024-03-25T16:13:14.267 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:49 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:14.429 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:14.432 INFO:teuthology.orchestra.run.smithi111.stdout:========================================================================================= 2024-03-25T16:13:14.432 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:13:14.432 INFO:teuthology.orchestra.run.smithi111.stdout:========================================================================================= 2024-03-25T16:13:14.432 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:13:14.432 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:13:14.432 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:13:14.432 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:13:14.432 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:13:14.433 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:13:14.434 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout:========================================================================================= 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout:Install 36 Packages 2024-03-25T16:13:14.435 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:14.436 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 30 M 2024-03-25T16:13:14.436 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 107 M 2024-03-25T16:13:14.436 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:13:15.056 INFO:teuthology.orchestra.run.smithi111.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-03-25T16:13:15.295 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:13:15.295 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:13:15.295 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:13:15.295 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:13:15.295 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:13:15.295 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:13:15.306 INFO:teuthology.orchestra.run.smithi111.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 1.5 MB 00:00 2024-03-25T16:13:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:13:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:15.616 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:13:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:13:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:13:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:13:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:13:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:15.617 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:15.902 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph 2024-03-25T16:13:15.982 INFO:teuthology.orchestra.run.smithi111.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2024-03-25T16:13:16.157 INFO:teuthology.orchestra.run.smithi111.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.4 MB/s | 247 kB 00:00 2024-03-25T16:13:16.374 INFO:teuthology.orchestra.run.smithi111.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 863 kB/s | 187 kB 00:00 2024-03-25T16:13:16.433 INFO:teuthology.orchestra.run.smithi111.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T16:13:16.466 INFO:teuthology.orchestra.run.smithi111.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-25T16:13:16.494 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:00:48 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:16.508 INFO:teuthology.orchestra.run.smithi111.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T16:13:16.558 INFO:teuthology.orchestra.run.smithi111.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-25T16:13:16.634 INFO:teuthology.orchestra.run.smithi111.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-25T16:13:16.668 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout:========================================================================================= 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout:========================================================================================= 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:13:16.671 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:13:16.672 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:13:16.673 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:13:16.674 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:13:16.674 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:16.674 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:13:16.674 INFO:teuthology.orchestra.run.smithi044.stdout:========================================================================================= 2024-03-25T16:13:16.674 INFO:teuthology.orchestra.run.smithi044.stdout:Install 36 Packages 2024-03-25T16:13:16.674 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:16.675 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 30 M 2024-03-25T16:13:16.675 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 107 M 2024-03-25T16:13:16.675 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:13:16.859 INFO:teuthology.orchestra.run.smithi111.stdout:(11/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 11 MB/s | 17 MB 00:01 2024-03-25T16:13:16.901 INFO:teuthology.orchestra.run.smithi111.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 4.3 MB/s | 1.1 MB 00:00 2024-03-25T16:13:16.926 INFO:teuthology.orchestra.run.smithi111.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-25T16:13:17.002 INFO:teuthology.orchestra.run.smithi111.stdout:(14/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.4 MB/s | 4.7 MB 00:01 2024-03-25T16:13:17.027 INFO:teuthology.orchestra.run.smithi111.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 990 kB/s | 124 kB 00:00 2024-03-25T16:13:17.052 INFO:teuthology.orchestra.run.smithi111.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.5 MB/s | 187 kB 00:00 2024-03-25T16:13:17.186 INFO:teuthology.orchestra.run.smithi111.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 682 kB/s | 125 kB 00:00 2024-03-25T16:13:17.211 INFO:teuthology.orchestra.run.smithi111.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-25T16:13:17.233 INFO:teuthology.orchestra.run.smithi044.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 33 kB/s | 6.6 kB 00:00 2024-03-25T16:13:17.745 INFO:teuthology.orchestra.run.smithi111.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 78 kB/s | 43 kB 00:00 2024-03-25T16:13:17.795 INFO:teuthology.orchestra.run.smithi111.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 204 kB/s | 151 kB 00:00 2024-03-25T16:13:17.912 INFO:teuthology.orchestra.run.smithi111.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-03-25T16:13:17.954 INFO:teuthology.orchestra.run.smithi111.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 231 kB/s | 172 kB 00:00 2024-03-25T16:13:18.021 INFO:teuthology.orchestra.run.smithi111.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-25T16:13:18.063 INFO:teuthology.orchestra.run.smithi111.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-25T16:13:18.129 INFO:teuthology.orchestra.run.smithi111.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-25T16:13:18.172 INFO:teuthology.orchestra.run.smithi111.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-25T16:13:18.240 INFO:teuthology.orchestra.run.smithi111.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 426 kB/s | 46 kB 00:00 2024-03-25T16:13:18.272 INFO:teuthology.orchestra.run.smithi111.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 680 kB/s | 358 kB 00:00 2024-03-25T16:13:18.300 INFO:teuthology.orchestra.run.smithi044.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-25T16:13:18.347 INFO:teuthology.orchestra.run.smithi111.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 450 kB/s | 79 kB 00:00 2024-03-25T16:13:18.373 INFO:teuthology.orchestra.run.smithi111.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-25T16:13:18.473 INFO:teuthology.orchestra.run.smithi111.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 358 kB/s | 36 kB 00:00 2024-03-25T16:13:18.523 INFO:teuthology.orchestra.run.smithi111.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 512 kB/s | 90 kB 00:00 2024-03-25T16:13:18.673 INFO:teuthology.orchestra.run.smithi111.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 626 kB/s | 272 kB 00:00 2024-03-25T16:13:18.774 INFO:teuthology.orchestra.run.smithi111.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-25T16:13:18.824 INFO:teuthology.orchestra.run.smithi111.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 655 kB/s | 230 kB 00:00 2024-03-25T16:13:19.124 INFO:teuthology.orchestra.run.smithi111.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 711 kB/s | 427 kB 00:00 2024-03-25T16:13:19.129 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:19.129 INFO:teuthology.orchestra.run.smithi111.stdout:Total 6.3 MB/s | 30 MB 00:04 2024-03-25T16:13:19.155 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:13:19.198 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:13:19.199 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:13:19.359 INFO:teuthology.orchestra.run.smithi044.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 935 kB/s | 2.1 MB 00:02 2024-03-25T16:13:19.526 INFO:teuthology.orchestra.run.smithi044.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.5 MB/s | 247 kB 00:00 2024-03-25T16:13:19.587 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:13:19.588 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:13:19.702 INFO:teuthology.orchestra.run.smithi044.stdout:(5/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 4.7 MB 00:02 2024-03-25T16:13:19.727 INFO:teuthology.orchestra.run.smithi044.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 934 kB/s | 187 kB 00:00 2024-03-25T16:13:19.760 INFO:teuthology.orchestra.run.smithi044.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T16:13:19.785 INFO:teuthology.orchestra.run.smithi044.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 540 kB/s | 32 kB 00:00 2024-03-25T16:13:19.811 INFO:teuthology.orchestra.run.smithi044.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-25T16:13:19.836 INFO:teuthology.orchestra.run.smithi044.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2024-03-25T16:13:19.869 INFO:teuthology.orchestra.run.smithi044.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T16:13:20.062 INFO:teuthology.orchestra.run.smithi044.stdout:(12/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 9.5 MB/s | 17 MB 00:01 2024-03-25T16:13:20.096 INFO:teuthology.orchestra.run.smithi044.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 4.4 MB/s | 1.1 MB 00:00 2024-03-25T16:13:20.120 INFO:teuthology.orchestra.run.smithi044.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 411 kB/s | 103 kB 00:00 2024-03-25T16:13:20.146 INFO:teuthology.orchestra.run.smithi044.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-25T16:13:20.171 INFO:teuthology.orchestra.run.smithi044.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T16:13:20.274 INFO:teuthology.orchestra.run.smithi044.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2024-03-25T16:13:20.613 INFO:teuthology.orchestra.run.smithi044.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-03-25T16:13:20.638 INFO:teuthology.orchestra.run.smithi044.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 324 kB/s | 151 kB 00:00 2024-03-25T16:13:20.925 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:13:20.973 INFO:teuthology.orchestra.run.smithi044.stdout:(20/36): python3-requests-2.25.1-8.el9.noarch.r 147 kB/s | 125 kB 00:00 2024-03-25T16:13:20.985 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:13:21.029 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:13:21.039 INFO:teuthology.orchestra.run.smithi044.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 404 kB/s | 172 kB 00:00 2024-03-25T16:13:21.054 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:13:21.117 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:13:21.156 INFO:teuthology.orchestra.run.smithi044.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-03-25T16:13:21.178 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:13:21.211 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:13:21.231 INFO:teuthology.orchestra.run.smithi044.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 605 kB/s | 358 kB 00:00 2024-03-25T16:13:21.299 INFO:teuthology.orchestra.run.smithi044.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 68 kB/s | 18 kB 00:00 2024-03-25T16:13:21.302 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:13:21.373 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:13:21.422 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:13:21.423 INFO:teuthology.orchestra.run.smithi044.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 87 kB/s | 23 kB 00:00 2024-03-25T16:13:21.494 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:13:21.507 INFO:teuthology.orchestra.run.smithi044.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 71 kB/s | 19 kB 00:00 2024-03-25T16:13:21.548 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:13:21.557 INFO:teuthology.orchestra.run.smithi044.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 76 kB/s | 20 kB 00:00 2024-03-25T16:13:21.586 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:13:21.635 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:13:21.692 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:13:21.774 INFO:teuthology.orchestra.run.smithi044.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 132 kB/s | 46 kB 00:00 2024-03-25T16:13:21.879 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:13:21.926 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:13:21.932 INFO:teuthology.orchestra.run.smithi044.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 185 kB/s | 79 kB 00:00 2024-03-25T16:13:21.956 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:13:21.990 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:13:22.029 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:13:22.032 INFO:teuthology.orchestra.run.smithi044.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 63 kB/s | 16 kB 00:00 2024-03-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi044.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 544 kB/s | 272 kB 00:00 2024-03-25T16:13:22.105 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:13:22.139 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:13:22.173 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:13:22.213 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:13:22.248 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:13:22.275 INFO:teuthology.orchestra.run.smithi044.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 263 kB/s | 90 kB 00:00 2024-03-25T16:13:22.283 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:13:22.300 INFO:teuthology.orchestra.run.smithi044.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-03-25T16:13:22.373 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:13:22.414 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:13:22.485 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:13:22.567 INFO:teuthology.orchestra.run.smithi044.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2024-03-25T16:13:22.592 INFO:teuthology.orchestra.run.smithi044.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 68 kB/s | 20 kB 00:00 2024-03-25T16:13:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:13:22.759 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:13:22.859 INFO:teuthology.orchestra.run.smithi044.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 731 kB/s | 427 kB 00:00 2024-03-25T16:13:22.863 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:22.863 INFO:teuthology.orchestra.run.smithi044.stdout:Total 4.8 MB/s | 30 MB 00:06 2024-03-25T16:13:22.889 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:13:22.922 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:13:22.936 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:13:22.936 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:13:22.952 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:13:22.977 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:13:22.977 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:22.977 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:13:22.978 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:13:22.978 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:13:22.978 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:23.350 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:13:23.351 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:13:24.403 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:13:24.429 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:13:24.429 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:24.429 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:13:24.429 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:13:24.429 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:13:24.429 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:24.685 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:13:24.762 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:13:24.796 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:13:24.823 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:13:24.870 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:13:24.893 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:13:24.893 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:24.893 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:13:24.893 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:13:24.893 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:13:24.893 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:24.897 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:13:24.938 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:13:24.971 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:13:25.067 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:13:25.113 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:13:25.135 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:13:25.136 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:25.136 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:13:25.136 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:13:25.136 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:13:25.136 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:25.152 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:13:25.200 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:13:25.201 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:13:25.274 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:13:25.326 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:13:25.371 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:13:25.430 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:13:25.512 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:13:25.687 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:13:25.734 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:13:25.780 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:13:25.815 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:13:25.845 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:13:25.924 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:13:25.964 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:13:25.997 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:13:26.037 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:13:26.072 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:13:26.107 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:13:26.199 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:13:26.246 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:13:26.322 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:13:26.469 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:13:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:13:26.770 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:13:26.792 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:13:26.817 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:13:26.818 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:26.818 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:13:26.818 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:13:26.818 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:13:26.818 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:13:27.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:13:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:13:27.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:13:27.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:13:27.939 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:27.939 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:13:27.940 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:13:27.941 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:27.942 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:28.296 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base 2024-03-25T16:13:28.334 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:13:28.360 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:13:28.360 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:28.360 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:13:28.360 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:13:28.360 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:13:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:28.835 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:13:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:13:28.858 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:28.858 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:13:28.858 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:13:28.858 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:13:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:28.865 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:03 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:28.960 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:29.012 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:29.013 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:13:29.013 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:29.072 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install cephadm 2024-03-25T16:13:29.077 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:13:29.100 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:13:29.100 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:29.100 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:13:29.101 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:13:29.101 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:13:29.101 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:29.168 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:13:29.623 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:04 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:29.771 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:29.772 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:29.772 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:13:29.772 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:29.772 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:13:29.772 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:13:29.772 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:29.773 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:13:29.773 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:29.773 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:13:29.773 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:29.773 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 747 k 2024-03-25T16:13:29.773 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 754 k 2024-03-25T16:13:29.773 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:13:30.078 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.4 MB/s | 747 kB 00:00 2024-03-25T16:13:30.079 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:30.079 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.4 MB/s | 747 kB 00:00 2024-03-25T16:13:30.079 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:13:30.084 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:13:30.084 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:13:30.089 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:13:30.090 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:13:30.163 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:13:30.561 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:30.648 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:13:30.885 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:13:30.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:13:30.887 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:13:30.887 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:13:30.887 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:13:30.887 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:13:30.887 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:13:30.887 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:13:30.887 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:13:30.888 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:13:30.888 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:13:30.889 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:13:30.889 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:13:30.889 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:13:30.889 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:13:30.889 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:13:30.889 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:13:31.032 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:31.297 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:31.297 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:31.297 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:13:31.298 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:31.298 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:31.298 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:31.505 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:13:31.871 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:13:31.871 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:13:31.872 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:13:31.873 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:13:31.874 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:13:31.875 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:13:31.875 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:31.875 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:32.069 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:07 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:32.215 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-base 2024-03-25T16:13:32.224 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 145 k 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 436 k 2024-03-25T16:13:32.225 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:13:32.593 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 397 kB/s | 145 kB 00:00 2024-03-25T16:13:32.593 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:32.594 INFO:teuthology.orchestra.run.smithi111.stdout:Total 395 kB/s | 145 kB 00:00 2024-03-25T16:13:32.594 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:13:32.601 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:13:32.601 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:13:32.653 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:13:32.653 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:13:32.788 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:04 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:32.792 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:13:32.860 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:32.884 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:32.884 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:32.884 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:13:32.885 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:32.888 INFO:teuthology.orchestra.run.smithi044.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:32.944 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:32.945 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:13:32.945 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:33.014 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install cephadm 2024-03-25T16:13:33.587 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:05 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:33.743 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:33.743 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 747 k 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 754 k 2024-03-25T16:13:33.744 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:13:33.855 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:33.855 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:33.855 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:13:33.855 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:33.856 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:33.856 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:34.092 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr 2024-03-25T16:13:34.664 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:09 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:34.684 INFO:teuthology.orchestra.run.smithi044.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 795 kB/s | 747 kB 00:00 2024-03-25T16:13:34.685 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:34.686 INFO:teuthology.orchestra.run.smithi044.stdout:Total 794 kB/s | 747 kB 00:00 2024-03-25T16:13:34.686 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:13:34.692 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:13:34.692 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:13:34.698 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:13:34.699 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:13:34.762 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:34.776 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:13:34.814 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:34.815 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:13:34.815 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:34.873 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:13:35.189 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:35.283 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:35.428 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:10 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:35.577 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:35.578 INFO:teuthology.orchestra.run.smithi111.stdout:====================================================================================== 2024-03-25T16:13:35.578 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:13:35.578 INFO:teuthology.orchestra.run.smithi111.stdout:====================================================================================== 2024-03-25T16:13:35.578 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:13:35.579 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:35.580 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:13:35.580 INFO:teuthology.orchestra.run.smithi111.stdout:====================================================================================== 2024-03-25T16:13:35.580 INFO:teuthology.orchestra.run.smithi111.stdout:Install 12 Packages 2024-03-25T16:13:35.580 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:35.580 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 6.9 M 2024-03-25T16:13:35.580 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2024-03-25T16:13:35.581 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:13:35.662 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:35.911 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:35.911 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:35.911 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:13:35.912 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:35.912 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:35.912 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:36.071 INFO:teuthology.orchestra.run.smithi111.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 109 kB/s | 31 kB 00:00 2024-03-25T16:13:36.106 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:13:36.121 INFO:teuthology.orchestra.run.smithi111.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 45 kB/s | 15 kB 00:00 2024-03-25T16:13:36.205 INFO:teuthology.orchestra.run.smithi111.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T16:13:36.264 INFO:teuthology.orchestra.run.smithi111.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.2 MB/s | 1.0 MB 00:00 2024-03-25T16:13:36.688 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:08 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:36.830 INFO:teuthology.orchestra.run.smithi111.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 967 kB/s | 548 kB 00:00 2024-03-25T16:13:36.844 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:36.845 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:36.845 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:13:36.845 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:36.846 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:13:36.846 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:13:36.846 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:36.846 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:13:36.847 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:13:36.847 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:13:36.847 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:36.847 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 145 k 2024-03-25T16:13:36.848 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 436 k 2024-03-25T16:13:36.848 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:13:36.934 INFO:teuthology.orchestra.run.smithi111.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-03-25T16:13:37.306 INFO:teuthology.orchestra.run.smithi111.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 5.4 MB/s | 2.0 MB 00:00 2024-03-25T16:13:37.415 INFO:teuthology.orchestra.run.smithi111.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-25T16:13:37.424 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 251 kB/s | 145 kB 00:00 2024-03-25T16:13:37.424 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:37.425 INFO:teuthology.orchestra.run.smithi044.stdout:Total 250 kB/s | 145 kB 00:00 2024-03-25T16:13:37.425 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:13:37.433 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:13:37.433 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:13:37.489 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:13:37.489 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:13:37.515 INFO:teuthology.orchestra.run.smithi111.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-25T16:13:37.557 INFO:teuthology.orchestra.run.smithi111.stdout:(10/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.0 MB/s | 1.8 MB 00:01 2024-03-25T16:13:37.632 INFO:teuthology.orchestra.run.smithi111.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-25T16:13:37.633 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:13:37.683 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:37.707 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:37.707 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:37.707 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:13:37.707 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:38.049 INFO:teuthology.orchestra.run.smithi111.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 468 kB/s | 863 kB 00:01 2024-03-25T16:13:38.053 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:38.053 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-25T16:13:38.073 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:13:38.092 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:13:38.092 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:13:38.265 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:13:38.266 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:13:38.668 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:38.668 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:38.668 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:13:38.668 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:38.668 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:38.668 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:13:38.857 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:13:38.901 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:13:38.904 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr 2024-03-25T16:13:38.947 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:13:38.987 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:13:39.100 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:13:39.227 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:13:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:13:39.366 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:13:39.395 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:39.426 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:13:39.478 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:11 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:39.579 INFO:teuthology.orchestra.run.smithi044.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:39.633 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:39.634 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:13:39.634 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:39.669 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:13:39.698 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:13:39.710 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:40.269 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:12 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:40.424 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:40.425 INFO:teuthology.orchestra.run.smithi044.stdout:====================================================================================== 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout:====================================================================================== 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:13:40.426 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout:====================================================================================== 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout:Install 12 Packages 2024-03-25T16:13:40.427 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:40.428 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 6.9 M 2024-03-25T16:13:40.428 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 26 M 2024-03-25T16:13:40.428 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:13:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:13:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:13:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:13:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:13:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:13:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:13:40.588 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:13:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:13:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:13:40.589 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:13:40.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:13:40.934 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:40.934 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:13:40.934 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:13:40.934 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:40.934 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:40.935 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:41.136 INFO:teuthology.orchestra.run.smithi044.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 120 kB/s | 31 kB 00:00 2024-03-25T16:13:41.170 INFO:teuthology.orchestra.run.smithi044.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 52 kB/s | 15 kB 00:00 2024-03-25T16:13:41.194 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:13:41.337 INFO:teuthology.orchestra.run.smithi044.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T16:13:41.362 INFO:teuthology.orchestra.run.smithi044.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T16:13:41.546 INFO:teuthology.orchestra.run.smithi044.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.9 MB/s | 548 kB 00:00 2024-03-25T16:13:41.588 INFO:teuthology.orchestra.run.smithi044.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-03-25T16:13:41.621 INFO:teuthology.orchestra.run.smithi044.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-25T16:13:41.713 INFO:teuthology.orchestra.run.smithi044.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-03-25T16:13:41.747 INFO:teuthology.orchestra.run.smithi044.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 922 kB/s | 31 kB 00:00 2024-03-25T16:13:41.757 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:41.764 INFO:teuthology.orchestra.run.smithi044.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.0 MB/s | 144 kB 00:00 2024-03-25T16:13:41.798 INFO:teuthology.orchestra.run.smithi044.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-25T16:13:41.908 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:41.909 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:13:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:Install 12 Packages 2024-03-25T16:13:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:41.912 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 37 M 2024-03-25T16:13:41.912 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 209 M 2024-03-25T16:13:41.912 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:13:42.181 INFO:teuthology.orchestra.run.smithi044.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.4 MB/s | 1.8 MB 00:01 2024-03-25T16:13:42.184 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:42.185 INFO:teuthology.orchestra.run.smithi044.stdout:Total 3.9 MB/s | 6.9 MB 00:01 2024-03-25T16:13:42.204 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:13:42.205 INFO:teuthology.orchestra.run.smithi111.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-25T16:13:42.222 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:13:42.223 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:13:42.239 INFO:teuthology.orchestra.run.smithi111.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2024-03-25T16:13:42.281 INFO:teuthology.orchestra.run.smithi111.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 826 kB/s | 34 kB 00:00 2024-03-25T16:13:42.348 INFO:teuthology.orchestra.run.smithi111.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-25T16:13:42.403 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:13:42.403 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:13:42.790 INFO:teuthology.orchestra.run.smithi111.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.3 MB/s | 4.7 MB 00:00 2024-03-25T16:13:42.857 INFO:teuthology.orchestra.run.smithi111.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.4 MB/s | 4.8 MB 00:00 2024-03-25T16:13:42.882 INFO:teuthology.orchestra.run.smithi111.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:13:42.937 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:13:43.024 INFO:teuthology.orchestra.run.smithi111.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 803 kB 00:00 2024-03-25T16:13:43.032 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:13:43.067 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:13:43.099 INFO:teuthology.orchestra.run.smithi111.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-03-25T16:13:43.114 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:13:43.144 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:13:43.248 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:13:43.373 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:13:43.465 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:13:43.466 INFO:teuthology.orchestra.run.smithi111.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-03-25T16:13:43.491 INFO:teuthology.orchestra.run.smithi111.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 628 kB/s | 245 kB 00:00 2024-03-25T16:13:43.515 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:13:43.543 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:43.575 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:13:43.828 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:13:43.858 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:44.433 INFO:teuthology.orchestra.run.smithi111.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.1 MB/s | 7.4 MB 00:02 2024-03-25T16:13:44.484 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:44.484 INFO:teuthology.orchestra.run.smithi111.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-25T16:13:44.484 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:13:44.503 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:13:44.503 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:13:44.614 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:44.614 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:13:44.614 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:13:44.615 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:13:44.859 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:13:44.859 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:13:44.969 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:13:44.970 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:13:44.970 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:13:44.970 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:13:44.970 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:13:44.970 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:13:44.970 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:44.970 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:45.207 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:13:45.568 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:13:45.648 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:13:45.679 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:13:45.770 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:45.779 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:17 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:45.807 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:13:45.938 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:45.939 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-03-25T16:13:45.939 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:13:45.939 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:13:45.940 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:13:45.941 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:13:45.941 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:13:45.941 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:45.941 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:13:45.941 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-03-25T16:13:45.941 INFO:teuthology.orchestra.run.smithi044.stdout:Install 12 Packages 2024-03-25T16:13:45.941 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:45.942 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 37 M 2024-03-25T16:13:45.942 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 209 M 2024-03-25T16:13:45.942 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:13:45.984 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:46.118 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:13:46.404 INFO:teuthology.orchestra.run.smithi044.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 137 kB/s | 32 kB 00:00 2024-03-25T16:13:46.488 INFO:teuthology.orchestra.run.smithi044.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 205 kB/s | 17 kB 00:00 2024-03-25T16:13:46.538 INFO:teuthology.orchestra.run.smithi044.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-25T16:13:46.778 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:13:46.884 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:13:46.896 INFO:teuthology.orchestra.run.smithi044.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-03-25T16:13:48.112 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:13:48.130 INFO:teuthology.orchestra.run.smithi044.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.0 MB/s | 4.7 MB 00:01 2024-03-25T16:13:48.230 INFO:teuthology.orchestra.run.smithi044.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.6 MB/s | 4.8 MB 00:01 2024-03-25T16:13:48.255 INFO:teuthology.orchestra.run.smithi044.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-25T16:13:48.439 INFO:teuthology.orchestra.run.smithi044.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-25T16:13:48.489 INFO:teuthology.orchestra.run.smithi044.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-25T16:13:48.673 INFO:teuthology.orchestra.run.smithi044.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-25T16:13:48.766 INFO:teuthology.orchestra.run.smithi044.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 2.9 MB/s | 7.4 MB 00:02 2024-03-25T16:13:48.790 INFO:teuthology.orchestra.run.smithi044.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 824 kB/s | 245 kB 00:00 2024-03-25T16:13:48.793 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:48.793 INFO:teuthology.orchestra.run.smithi044.stdout:Total 13 MB/s | 37 MB 00:02 2024-03-25T16:13:48.834 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:13:48.858 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:13:48.858 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:13:49.139 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:13:49.163 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:49.234 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:13:49.234 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:13:49.961 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:13:50.047 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:13:50.078 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:13:50.176 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:50.215 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:13:50.400 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:50.430 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:50.518 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:13:51.217 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:13:51.339 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:13:51.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:13:51.862 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:13:52.307 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:13:52.307 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:52.307 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:13:52.307 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:52.307 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:13:52.307 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:13:52.307 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:13:52.307 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:52.308 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:52.648 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:13:52.747 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:13:53.316 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:28 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:53.466 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:53.467 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:13:53.467 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:13:53.467 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:13:53.467 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:13:53.467 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:53.468 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:13:53.469 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:13:53.469 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-03-25T16:13:53.469 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:53.469 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 2.0 M 2024-03-25T16:13:53.469 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2024-03-25T16:13:53.469 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:13:53.656 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 223 kB/s | 24 kB 00:00 2024-03-25T16:13:53.714 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 108 kB/s | 18 kB 00:00 2024-03-25T16:13:53.729 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:13:53.754 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:53.764 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-25T16:13:53.789 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:13:53.823 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:13:53.840 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 865 kB/s | 43 kB 00:00 2024-03-25T16:13:53.965 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-03-25T16:13:53.991 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 193 kB/s | 32 kB 00:00 2024-03-25T16:13:54.134 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.2 MB/s | 203 kB 00:00 2024-03-25T16:13:54.150 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 87 kB/s | 52 kB 00:00 2024-03-25T16:13:54.184 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-25T16:13:54.250 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-25T16:13:54.309 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-03-25T16:13:54.310 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:54.311 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-25T16:13:54.311 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:13:54.324 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:13:54.324 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:13:54.414 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:13:54.415 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:13:54.920 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:13:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:13:55.143 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:13:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:13:55.244 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:13:55.285 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:13:55.380 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:13:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:13:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:13:56.262 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:13:56.298 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:56.298 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:13:56.298 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:13:56.298 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:56.298 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:13:56.298 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:56.299 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:56.299 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:13:56.299 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:13:56.299 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:13:56.299 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:13:56.299 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:13:56.309 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:13:56.351 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:13:56.400 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:13:56.421 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:13:56.735 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:13:56.735 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:56.735 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:13:56.735 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:13:57.111 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:13:57.199 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:13:57.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:13:57.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:13:57.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:13:57.200 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:13:57.557 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:13:57.558 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:13:57.558 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:13:57.558 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:13:57.558 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:13:57.558 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:13:57.558 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:57.558 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:13:57.689 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:29 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:13:57.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:13:57.845 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:13:57.846 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:13:57.846 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:13:57.846 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:13:57.846 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:13:57.847 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:13:57.848 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:13:57.848 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:57.848 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:13:57.848 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:13:57.848 INFO:teuthology.orchestra.run.smithi044.stdout:Install 13 Packages 2024-03-25T16:13:57.848 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:13:57.848 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 2.0 M 2024-03-25T16:13:57.848 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 26 M 2024-03-25T16:13:57.849 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:13:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 179 kB/s | 18 kB 00:00 2024-03-25T16:13:58.063 INFO:teuthology.orchestra.run.smithi044.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 145 kB/s | 24 kB 00:00 2024-03-25T16:13:58.088 INFO:teuthology.orchestra.run.smithi044.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T16:13:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-25T16:13:58.139 INFO:teuthology.orchestra.run.smithi044.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-25T16:13:58.164 INFO:teuthology.orchestra.run.smithi044.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-25T16:13:58.378 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:33 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:13:58.381 INFO:teuthology.orchestra.run.smithi044.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 107 kB/s | 52 kB 00:00 2024-03-25T16:13:58.528 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:13:58.529 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:13:58.529 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:13:58.529 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:13:58.529 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:13:58.530 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:13:58.531 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-03-25T16:13:58.531 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:13:58.531 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 6.8 M 2024-03-25T16:13:58.531 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 33 M 2024-03-25T16:13:58.531 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:13:58.623 INFO:teuthology.orchestra.run.smithi044.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 66 kB/s | 32 kB 00:00 2024-03-25T16:13:58.648 INFO:teuthology.orchestra.run.smithi044.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 31 kB/s | 15 kB 00:00 2024-03-25T16:13:58.840 INFO:teuthology.orchestra.run.smithi044.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-03-25T16:13:59.032 INFO:teuthology.orchestra.run.smithi044.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 468 kB/s | 90 kB 00:00 2024-03-25T16:13:59.158 INFO:teuthology.orchestra.run.smithi044.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 262 kB/s | 203 kB 00:00 2024-03-25T16:13:59.493 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-25T16:13:59.526 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 633 kB/s | 248 kB 00:00 2024-03-25T16:13:59.710 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 315 kB/s | 58 kB 00:00 2024-03-25T16:13:59.769 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 872 kB/s | 51 kB 00:00 2024-03-25T16:13:59.852 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T16:13:59.886 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 183 kB/s | 138 kB 00:00 2024-03-25T16:13:59.887 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:59.887 INFO:teuthology.orchestra.run.smithi111.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2024-03-25T16:13:59.900 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:13:59.911 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:13:59.911 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:13:59.999 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:13:59.999 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:00.091 INFO:teuthology.orchestra.run.smithi044.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 712 kB/s | 1.0 MB 00:01 2024-03-25T16:14:00.096 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:00.097 INFO:teuthology.orchestra.run.smithi044.stdout:Total 933 kB/s | 2.0 MB 00:02 2024-03-25T16:14:00.097 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:00.113 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:00.113 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:00.215 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:00.216 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:00.299 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:14:00.470 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:14:00.521 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:14:00.743 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:00.912 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:14:00.968 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:14:01.024 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:14:01.041 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:14:01.060 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:14:01.102 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:14:01.140 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:14:01.218 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:14:01.218 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:14:01.225 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:14:01.309 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:14:01.371 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:14:02.003 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:14:02.004 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:14:02.004 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:14:02.004 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:14:02.004 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:14:02.004 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:14:02.131 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:14:02.183 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:14:02.217 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:14:02.266 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:14:02.295 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:14:02.296 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:02.297 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:02.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse 2024-03-25T16:14:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:14:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:14:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:14:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:14:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:14:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:14:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:14:03.056 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:14:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:14:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:14:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:14:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:14:03.057 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:14:03.105 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:38 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:14:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:03.258 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-03-25T16:14:03.258 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:03.258 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 964 k 2024-03-25T16:14:03.258 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.8 M 2024-03-25T16:14:03.258 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:14:03.396 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 574 kB/s | 79 kB 00:00 2024-03-25T16:14:03.414 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:14:03.415 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:14:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:14:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:14:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:14:03.416 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:03.416 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:03.656 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:14:04.226 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:36 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:04.384 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:14:04.385 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:14:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:14:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:04.386 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:04.386 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:14:04.386 INFO:teuthology.orchestra.run.smithi044.stdout:Install 6 Packages 2024-03-25T16:14:04.386 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:04.386 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 6.8 M 2024-03-25T16:14:04.387 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 33 M 2024-03-25T16:14:04.387 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:04.446 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 745 kB/s | 885 kB 00:01 2024-03-25T16:14:04.447 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:04.448 INFO:teuthology.orchestra.run.smithi111.stdout:Total 810 kB/s | 964 kB 00:01 2024-03-25T16:14:04.448 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:14:04.471 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:14:04.471 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:14:04.547 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:14:04.548 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:04.732 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:14:04.880 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:14:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:14:05.268 INFO:teuthology.orchestra.run.smithi044.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-25T16:14:05.580 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:14:05.581 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:14:05.651 INFO:teuthology.orchestra.run.smithi044.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 176 kB/s | 138 kB 00:00 2024-03-25T16:14:05.872 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:14:05.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:05.872 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:14:05.872 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:14:05.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:05.872 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:05.976 INFO:teuthology.orchestra.run.smithi044.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 775 kB/s | 548 kB 00:00 2024-03-25T16:14:06.002 INFO:teuthology.orchestra.run.smithi044.stdout:(4/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 218 kB/s | 248 kB 00:01 2024-03-25T16:14:06.019 INFO:teuthology.orchestra.run.smithi044.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 157 kB/s | 58 kB 00:00 2024-03-25T16:14:06.146 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-volume 2024-03-25T16:14:06.210 INFO:teuthology.orchestra.run.smithi044.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 218 kB/s | 51 kB 00:00 2024-03-25T16:14:06.212 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:06.212 INFO:teuthology.orchestra.run.smithi044.stdout:Total 3.7 MB/s | 6.8 MB 00:01 2024-03-25T16:14:06.227 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:06.238 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:06.239 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:06.333 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:06.334 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:06.643 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:06.722 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:41 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:06.832 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:14:06.880 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:06.881 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-03-25T16:14:06.882 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:06.882 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 332 k 2024-03-25T16:14:06.882 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.5 M 2024-03-25T16:14:06.882 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:14:06.980 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:14:07.150 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 257 kB/s | 69 kB 00:00 2024-03-25T16:14:07.528 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:14:07.609 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:14:07.677 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:14:07.707 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:14:07.959 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 245 kB/s | 264 kB 00:01 2024-03-25T16:14:07.960 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:07.960 INFO:teuthology.orchestra.run.smithi111.stdout:Total 308 kB/s | 332 kB 00:01 2024-03-25T16:14:07.960 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:14:07.970 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:14:07.970 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:14:08.013 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:14:08.013 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:08.162 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:14:08.341 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:14:08.395 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:14:08.410 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:14:08.410 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:14:08.410 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:14:08.410 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:14:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:14:08.502 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:14:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:14:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:14:08.503 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:14:08.807 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:08.808 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:09.049 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-fuse 2024-03-25T16:14:09.149 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:14:09.493 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:14:09.493 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:09.493 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:14:09.493 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:14:09.493 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:14:09.494 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:09.494 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:09.628 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:41 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:09.785 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:14:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:14:09.786 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:14:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:14:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:09.787 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:09.787 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:09.787 INFO:teuthology.orchestra.run.smithi044.stdout:Install 2 Packages 2024-03-25T16:14:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:09.787 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 964 k 2024-03-25T16:14:09.787 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.8 M 2024-03-25T16:14:09.788 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:09.817 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel 2024-03-25T16:14:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 559 kB/s | 79 kB 00:00 2024-03-25T16:14:10.401 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:10.550 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:10.551 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:14:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:14:10.551 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:14:10.551 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:14:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:14:10.552 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 127 k 2024-03-25T16:14:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 456 k 2024-03-25T16:14:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:14:10.845 INFO:teuthology.orchestra.run.smithi044.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 837 kB/s | 885 kB 00:01 2024-03-25T16:14:10.846 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:10.847 INFO:teuthology.orchestra.run.smithi044.stdout:Total 909 kB/s | 964 kB 00:01 2024-03-25T16:14:10.847 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:10.871 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:10.871 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:10.951 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:10.952 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:11.139 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:11.181 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 201 kB/s | 127 kB 00:00 2024-03-25T16:14:11.182 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:11.182 INFO:teuthology.orchestra.run.smithi111.stdout:Total 201 kB/s | 127 kB 00:00 2024-03-25T16:14:11.182 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:14:11.188 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:14:11.188 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:14:11.246 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:14:11.246 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:11.267 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:14:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:14:11.386 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:14:11.496 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:11.972 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:12.049 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:14:12.050 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:14:12.279 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:12.279 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:12.279 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:14:12.279 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:12.279 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:12.279 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:12.332 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:14:12.332 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:12.332 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:12.332 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:14:12.332 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:12.333 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:12.502 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2 2024-03-25T16:14:12.594 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-volume 2024-03-25T16:14:13.072 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:13.170 INFO:teuthology.orchestra.run.smithi111.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:13.175 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:13.223 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:13.224 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:14:13.224 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:13.294 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs-devel 2024-03-25T16:14:13.333 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:13.334 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:13.334 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:13.335 INFO:teuthology.orchestra.run.smithi044.stdout:Install 2 Packages 2024-03-25T16:14:13.336 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:13.336 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 332 k 2024-03-25T16:14:13.336 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 1.5 M 2024-03-25T16:14:13.336 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:13.452 INFO:teuthology.orchestra.run.smithi044.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 592 kB/s | 69 kB 00:00 2024-03-25T16:14:13.858 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:14.005 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:14.007 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 31 k 2024-03-25T16:14:14.008 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 140 k 2024-03-25T16:14:14.008 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:14:14.110 INFO:teuthology.orchestra.run.smithi044.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 340 kB/s | 264 kB 00:00 2024-03-25T16:14:14.111 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:14.112 INFO:teuthology.orchestra.run.smithi044.stdout:Total 428 kB/s | 332 kB 00:00 2024-03-25T16:14:14.112 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:14.122 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:14.122 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:14.159 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:14.160 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:14.315 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:14.392 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 81 kB/s | 31 kB 00:00 2024-03-25T16:14:14.393 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:14.393 INFO:teuthology.orchestra.run.smithi111.stdout:Total 81 kB/s | 31 kB 00:00 2024-03-25T16:14:14.393 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:14:14.396 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:14:14.397 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:14:14.421 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:14:14.421 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:14.491 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:14:14.521 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:14:14.536 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:14:14.537 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:14:14.537 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:14:14.537 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:14.548 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:14:14.665 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:15.070 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:15.268 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:14:15.318 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:15.318 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:15.318 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:14:15.318 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:15.318 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:15.318 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:15.481 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2 2024-03-25T16:14:15.605 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:14:15.605 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:15.605 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:15.605 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:14:15.605 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:14:15.605 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:15.605 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:15.943 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados-devel 2024-03-25T16:14:16.043 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:51 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:16.140 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:16.190 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:16.191 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:14:16.192 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:16.241 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1 2024-03-25T16:14:16.522 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:16.677 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:16.677 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:16.677 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:14:16.677 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:16.677 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:16.678 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:14:16.678 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:16.678 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:16.678 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:16.678 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:14:16.678 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:16.678 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 127 k 2024-03-25T16:14:16.679 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 456 k 2024-03-25T16:14:16.679 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:16.803 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:51 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:16.899 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:16.951 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:16.952 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:14:16.952 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:16.995 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados 2024-03-25T16:14:17.208 INFO:teuthology.orchestra.run.smithi044.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 239 kB/s | 127 kB 00:00 2024-03-25T16:14:17.208 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:17.208 INFO:teuthology.orchestra.run.smithi044.stdout:Total 239 kB/s | 127 kB 00:00 2024-03-25T16:14:17.209 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:17.215 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:17.215 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:17.275 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:17.275 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:17.428 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:17.525 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:17.560 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:52 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:17.657 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:17.708 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:17.709 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:14:17.709 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:17.753 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw 2024-03-25T16:14:17.965 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:18.228 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:18.228 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:18.228 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:18.228 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:18.228 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:18.228 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:18.318 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:53 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:18.415 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:18.461 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install libcephfs2 2024-03-25T16:14:18.465 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:18.466 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:14:18.467 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:18.518 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs 2024-03-25T16:14:19.041 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:51 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:19.084 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:19.141 INFO:teuthology.orchestra.run.smithi044.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:19.180 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:19.197 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:19.198 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:14:19.198 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:19.231 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:19.232 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:14:19.232 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:19.264 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install libcephfs-devel 2024-03-25T16:14:19.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd 2024-03-25T16:14:19.833 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:19.841 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:51 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:19.928 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:19.979 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:19.980 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T16:14:19.980 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:19.996 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:14:19.997 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:19.998 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 31 k 2024-03-25T16:14:19.998 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 140 k 2024-03-25T16:14:19.998 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:20.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse 2024-03-25T16:14:20.427 INFO:teuthology.orchestra.run.smithi044.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 73 kB/s | 31 kB 00:00 2024-03-25T16:14:20.428 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:20.428 INFO:teuthology.orchestra.run.smithi044.stdout:Total 72 kB/s | 31 kB 00:00 2024-03-25T16:14:20.428 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:20.432 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:20.432 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:20.469 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:20.470 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:20.596 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:20.623 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:55 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:20.696 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:20.779 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:20.780 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:14:20.780 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:14:20.780 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:14:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:14:20.781 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 85 k 2024-03-25T16:14:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 231 k 2024-03-25T16:14:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:14:21.097 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:21.212 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 198 kB/s | 85 kB 00:00 2024-03-25T16:14:21.213 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:21.213 INFO:teuthology.orchestra.run.smithi111.stdout:Total 197 kB/s | 85 kB 00:00 2024-03-25T16:14:21.213 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:14:21.221 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:14:21.221 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:14:21.279 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:14:21.279 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:21.364 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:21.364 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:21.364 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:21.412 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:14:21.511 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:21.525 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados2 2024-03-25T16:14:21.899 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:22.103 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:22.168 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:22.168 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:22.168 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:14:22.168 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:22.168 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:22.168 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:22.204 INFO:teuthology.orchestra.run.smithi044.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:22.258 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:22.260 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:14:22.260 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:22.304 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librbd1 2024-03-25T16:14:22.397 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2024-03-25T16:14:22.881 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:22.979 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:57 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:22.982 INFO:teuthology.orchestra.run.smithi044.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:23.037 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:23.038 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:14:23.038 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:23.086 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rados 2024-03-25T16:14:23.127 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:23.127 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.1 M 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 13 M 2024-03-25T16:14:23.128 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:14:23.661 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:55 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:23.760 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:23.814 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:23.815 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:14:23.815 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:23.865 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rgw 2024-03-25T16:14:24.442 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:56 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:24.547 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:24.602 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:24.603 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:14:24.603 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:24.653 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-cephfs 2024-03-25T16:14:25.238 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:57 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:25.308 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.4 MB/s | 3.1 MB 00:02 2024-03-25T16:14:25.309 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:25.309 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.4 MB/s | 3.1 MB 00:02 2024-03-25T16:14:25.309 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:14:25.320 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:14:25.320 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:14:25.338 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:25.392 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:25.393 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:14:25.393 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:25.396 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:14:25.397 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:25.437 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rbd 2024-03-25T16:14:25.827 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:14:25.884 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:25.908 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:25.908 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:14:25.908 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:14:25.908 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:14:25.908 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:14:25.908 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:26.009 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:26.108 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:26.163 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:26.164 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:14:26.164 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:26.208 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-fuse 2024-03-25T16:14:26.785 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:26.939 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:26.940 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 85 k 2024-03-25T16:14:26.941 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 231 k 2024-03-25T16:14:26.941 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:27.033 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:14:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:27.033 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:27.033 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:27.288 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd 2024-03-25T16:14:27.489 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 155 kB/s | 85 kB 00:00 2024-03-25T16:14:27.490 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:27.490 INFO:teuthology.orchestra.run.smithi044.stdout:Total 155 kB/s | 85 kB 00:00 2024-03-25T16:14:27.490 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:27.498 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:27.498 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:27.556 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:27.557 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:27.694 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:27.782 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:27.865 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:28.013 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:28.014 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 171 k 2024-03-25T16:14:28.015 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 494 k 2024-03-25T16:14:28.015 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:14:28.171 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:28.472 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:28.473 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:28.473 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:28.473 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:28.697 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-mirror 2024-03-25T16:14:28.736 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 238 kB/s | 171 kB 00:00 2024-03-25T16:14:28.737 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:28.737 INFO:teuthology.orchestra.run.smithi111.stdout:Total 237 kB/s | 171 kB 00:00 2024-03-25T16:14:28.737 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:14:28.745 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:14:28.745 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:14:28.802 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:14:28.802 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:14:29.060 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:29.274 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:01 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:29.430 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:29.430 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:29.430 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:14:29.430 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 3.1 M 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 13 M 2024-03-25T16:14:29.431 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:29.732 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:29.733 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:29.733 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:14:29.733 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:29.733 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:29.733 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:14:29.945 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:14:30.516 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:14:30.668 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:14:30.669 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 91 M 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 498 M 2024-03-25T16:14:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:14:31.834 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.3 MB/s | 3.1 MB 00:02 2024-03-25T16:14:31.834 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:31.835 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.3 MB/s | 3.1 MB 00:02 2024-03-25T16:14:31.835 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:31.847 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:31.847 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:31.926 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:31.926 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:32.384 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:32.451 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:32.475 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:32.475 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:14:32.475 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:14:32.475 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:14:32.475 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:14:32.475 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:33.543 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:33.543 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:33.543 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:33.543 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:33.543 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:33.544 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:33.778 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-nbd 2024-03-25T16:14:34.358 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:06 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:34.515 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:34.516 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:14:34.516 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:14:34.516 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 171 k 2024-03-25T16:14:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 494 k 2024-03-25T16:14:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:34.978 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 372 kB/s | 171 kB 00:00 2024-03-25T16:14:34.979 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:34.980 INFO:teuthology.orchestra.run.smithi044.stdout:Total 370 kB/s | 171 kB 00:00 2024-03-25T16:14:34.980 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:34.989 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:34.989 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:35.046 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:35.047 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:35.199 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:35.305 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:35.723 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:36.004 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:36.004 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:36.004 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:14:36.004 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:36.004 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:36.004 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:14:36.214 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:14:36.793 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:08 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:14:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:14:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:36.949 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:14:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:14:36.949 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:14:36.949 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T16:14:36.949 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:14:36.950 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:36.950 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:14:36.950 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:14:36.950 INFO:teuthology.orchestra.run.smithi044.stdout:Install 2 Packages 2024-03-25T16:14:36.950 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:14:36.950 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 91 M 2024-03-25T16:14:36.951 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 498 M 2024-03-25T16:14:36.951 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:14:42.005 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 2.0 MB/s | 22 MB 00:11 2024-03-25T16:14:50.781 INFO:teuthology.orchestra.run.smithi044.stdout:(1/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 4.9 MB/s | 68 MB 00:13 2024-03-25T16:14:53.905 INFO:teuthology.orchestra.run.smithi044.stdout:(2/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 1.3 MB/s | 22 MB 00:16 2024-03-25T16:14:53.907 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:53.907 INFO:teuthology.orchestra.run.smithi044.stdout:Total 5.3 MB/s | 91 MB 00:16 2024-03-25T16:14:53.908 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:14:53.913 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:14:53.913 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:14:54.526 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:14:54.527 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:14:55.295 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 2.8 MB/s | 68 MB 00:24 2024-03-25T16:14:55.297 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:55.297 INFO:teuthology.orchestra.run.smithi111.stdout:Total 3.7 MB/s | 91 MB 00:24 2024-03-25T16:14:55.298 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:14:55.302 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:14:55.302 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:14:55.881 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:14:55.881 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:14:57.811 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:14:58.996 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:15:05.338 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:15:05.417 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:15:06.099 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:15:06.170 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:15:08.784 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:15:08.784 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:15:09.057 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:15:09.057 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:15:09.057 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:15:09.057 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:09.057 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:09.057 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:15:09.057 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:15:09.258 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:15:09.677 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:15:09.677 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:15:09.832 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:02:41 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:15:09.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:15:09.936 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:15:09.936 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:15:09.936 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:09.936 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:09.936 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:15:09.936 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:15:09.987 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:15:09.988 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:15:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:15:09.988 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 416 M 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 1.7 G 2024-03-25T16:15:09.989 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:15:10.163 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:15:10.737 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:45 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:15:10.897 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:15:10.897 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:15:10.898 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:15:10.899 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 416 M 2024-03-25T16:15:10.899 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.7 G 2024-03-25T16:15:10.899 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:17:43.378 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 2.7 MB/s | 416 MB 02:33 2024-03-25T16:17:43.379 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:43.379 INFO:teuthology.orchestra.run.smithi044.stdout:Total 2.7 MB/s | 416 MB 02:33 2024-03-25T16:17:43.379 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:17:43.384 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:17:43.384 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:17:45.973 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:17:45.974 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:18:09.277 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 2.3 MB/s | 416 MB 02:58 2024-03-25T16:18:09.277 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:09.277 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.3 MB/s | 416 MB 02:58 2024-03-25T16:18:09.278 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:18:09.282 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:18:09.282 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:18:11.743 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:18:11.744 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:18:30.697 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:18:30.809 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:34.440 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:34.705 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:34.705 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:18:34.705 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:18:34.705 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:34.705 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:18:34.705 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:18:34.957 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:18:35.533 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:06:07 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:18:35.687 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:18:35.688 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:18:35.688 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:18:35.688 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:18:35.688 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:18:35.688 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:18:35.688 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:18:35.688 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:18:35.689 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T16:18:35.689 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:18:35.689 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:18:35.689 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 1.4 M 2024-03-25T16:18:35.689 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 6.4 M 2024-03-25T16:18:35.689 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:18:37.094 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 1.0 MB/s | 1.4 MB 00:01 2024-03-25T16:18:37.095 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:37.095 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.0 MB/s | 1.4 MB 00:01 2024-03-25T16:18:37.095 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:18:37.099 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:18:37.099 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:18:37.146 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:18:37.146 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:18:37.418 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:18:37.529 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:18:37.891 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:18:38.160 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:18:38.161 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:18:38.161 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:18:38.161 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:38.161 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:18:38.161 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:18:38.338 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:18:38.914 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:18:39.068 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:18:39.069 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:18:39.070 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 176 M 2024-03-25T16:18:39.070 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 753 M 2024-03-25T16:18:39.070 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:18:53.781 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:18:53.887 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:58.938 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:59.189 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:59.189 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:18:59.189 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:18:59.189 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:59.189 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:18:59.189 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:18:59.434 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:19:00.003 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:06:35 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:19:00.151 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.4 M 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 6.4 M 2024-03-25T16:19:00.152 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:19:02.875 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 537 kB/s | 1.4 MB 00:02 2024-03-25T16:19:02.875 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:02.876 INFO:teuthology.orchestra.run.smithi111.stdout:Total 536 kB/s | 1.4 MB 00:02 2024-03-25T16:19:02.876 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:19:02.880 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:19:02.880 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:19:02.924 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:19:02.925 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:19:03.175 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:19:03.281 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:19:03.663 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:19:03.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:19:03.937 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:19:03.937 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:19:03.937 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:03.937 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:19:03.937 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:19:04.113 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:19:04.680 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:06:39 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:19:04.829 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:19:04.830 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:19:04.831 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 176 M 2024-03-25T16:19:04.831 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 753 M 2024-03-25T16:19:04.831 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:21:25.718 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 1.1 MB/s | 176 MB 02:46 2024-03-25T16:21:25.718 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.1 MB/s | 176 MB 02:46 2024-03-25T16:21:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:21:25.719 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:21:25.720 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:21:26.828 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:21:26.828 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:21:45.899 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:21:45.996 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:21:50.661 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:21:50.921 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:21:50.921 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:21:50.921 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:21:50.921 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:50.922 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:21:50.922 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:21:51.158 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:21:51.732 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:09:23 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:21:51.887 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:21:51.887 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 672 M 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 3.1 G 2024-03-25T16:21:51.888 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:22:32.076 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 868 kB/s | 176 MB 03:27 2024-03-25T16:22:32.077 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:32.077 INFO:teuthology.orchestra.run.smithi111.stdout:Total 868 kB/s | 176 MB 03:27 2024-03-25T16:22:32.077 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T16:22:32.080 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T16:22:32.080 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T16:22:33.141 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T16:22:33.141 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T16:22:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T16:22:51.464 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:22:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:22:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:22:56.378 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T16:22:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:22:56.378 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T16:22:56.619 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:22:57.191 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:10:32 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T16:22:57.351 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T16:22:57.352 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:22:57.352 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T16:22:57.352 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:22:57.352 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T16:22:57.352 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:22:57.353 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:22:57.353 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T16:22:57.353 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T16:22:57.353 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T16:22:57.353 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T16:22:57.353 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 672 M 2024-03-25T16:22:57.353 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 3.1 G 2024-03-25T16:22:57.353 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T16:45:35.767 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 483 kB/s | 672 MB 23:43 2024-03-25T16:45:35.794 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:35.795 INFO:teuthology.orchestra.run.smithi044.stdout:Total 483 kB/s | 672 MB 23:43 2024-03-25T16:45:35.795 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:45:35.795 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:45:35.795 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:45:39.923 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:45:39.924 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:46:54.882 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:46:54.988 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:47:04.190 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:47:04.458 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:47:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:47:04.459 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:47:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:47:04.459 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:47:04.459 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:47:04.739 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:47:05.316 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:34:37 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:47:05.417 INFO:teuthology.orchestra.run.smithi044.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:47:05.472 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:47:05.473 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T16:47:05.473 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:47:05.539 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T16:47:06.111 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:34:38 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:47:06.265 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:47:06.265 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:47:06.265 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 20 M 2024-03-25T16:47:06.266 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 103 M 2024-03-25T16:47:06.267 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:47:55.101 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 413 kB/s | 20 MB 00:48 2024-03-25T16:47:55.101 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:47:55.102 INFO:teuthology.orchestra.run.smithi044.stdout:Total 413 kB/s | 20 MB 00:48 2024-03-25T16:47:55.102 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:47:55.106 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:47:55.106 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:47:55.263 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:47:55.263 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:47:57.618 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:47:57.861 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:47:58.785 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:47:59.050 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:47:59.050 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:47:59.051 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:47:59.051 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:47:59.051 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:47:59.051 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:47:59.224 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T16:47:59.799 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:35:31 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:47:59.953 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:47:59.954 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:47:59.954 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:47:59.954 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:47:59.954 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:47:59.954 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T16:47:59.954 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:47:59.954 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:47:59.954 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:47:59.955 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:47:59.955 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:47:59.955 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 48 M 2024-03-25T16:47:59.955 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 173 M 2024-03-25T16:47:59.955 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:51:07.733 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 260 kB/s | 48 MB 03:07 2024-03-25T16:51:07.734 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:51:07.734 INFO:teuthology.orchestra.run.smithi044.stdout:Total 260 kB/s | 48 MB 03:07 2024-03-25T16:51:07.734 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:51:07.738 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:51:07.738 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:51:08.064 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:51:08.064 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:51:13.097 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:51:13.193 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:51:14.559 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:51:14.842 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:51:14.843 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:51:14.843 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:51:14.843 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:51:14.843 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:51:14.843 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:51:15.027 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T16:51:15.600 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:38:47 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:51:15.753 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:51:15.754 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:51:15.754 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:51:15.754 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:51:15.754 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:51:15.754 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T16:51:15.755 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:51:15.755 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:51:15.755 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:51:15.755 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:51:15.755 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:51:15.755 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 99 M 2024-03-25T16:51:15.755 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 449 M 2024-03-25T16:51:15.755 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T16:56:17.637 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 334 kB/s | 99 MB 05:01 2024-03-25T16:56:17.637 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:17.638 INFO:teuthology.orchestra.run.smithi044.stdout:Total 334 kB/s | 99 MB 05:01 2024-03-25T16:56:17.638 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T16:56:17.641 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T16:56:17.641 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T16:56:18.280 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T16:56:18.280 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T16:56:29.177 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T16:56:29.270 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:56:32.178 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:56:32.429 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:56:32.429 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:56:32.429 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T16:56:32.429 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:32.430 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:56:32.430 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T16:56:32.605 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T16:56:33.180 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:44:05 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T16:56:33.336 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T16:56:33.337 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:56:33.337 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T16:56:33.337 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 287 M 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 1.2 G 2024-03-25T16:56:33.338 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:00:21.362 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 307 kB/s | 672 MB 37:24 2024-03-25T17:00:21.362 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:00:21.363 INFO:teuthology.orchestra.run.smithi111.stdout:Total 307 kB/s | 672 MB 37:24 2024-03-25T17:00:21.363 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:00:21.368 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:00:21.368 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:00:25.319 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:00:25.319 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:01:35.742 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:01:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:01:45.862 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:01:46.127 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:01:46.127 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:01:46.127 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:01:46.128 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:01:46.128 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:01:46.128 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:01:46.423 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:01:46.987 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:49:21 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:01:47.083 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:01:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:01:47.137 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T17:01:47.137 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:01:47.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:01:47.746 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:49:22 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:01:47.892 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 20 M 2024-03-25T17:01:47.893 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 103 M 2024-03-25T17:01:47.894 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:03:06.655 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 256 kB/s | 20 MB 01:18 2024-03-25T17:03:06.656 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:03:06.657 INFO:teuthology.orchestra.run.smithi111.stdout:Total 256 kB/s | 20 MB 01:18 2024-03-25T17:03:06.657 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:03:06.660 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:03:06.660 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:03:06.809 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:03:06.809 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:03:09.029 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:03:09.127 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:03:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:03:10.356 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:03:10.356 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:03:10.357 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:03:10.357 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:03:10.357 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:03:10.357 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:03:10.515 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:03:11.074 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:50:46 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:03:11.222 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:03:11.223 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:03:11.224 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 48 M 2024-03-25T17:03:11.224 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 173 M 2024-03-25T17:03:11.224 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:06:09.283 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 274 kB/s | 48 MB 02:58 2024-03-25T17:06:09.284 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:06:09.285 INFO:teuthology.orchestra.run.smithi111.stdout:Total 274 kB/s | 48 MB 02:58 2024-03-25T17:06:09.285 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:06:09.287 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:06:09.288 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:06:09.599 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:06:09.599 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:06:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:06:14.490 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:06:15.887 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:06:16.154 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:06:16.154 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:06:16.154 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:06:16.154 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:06:16.154 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:06:16.154 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:06:16.348 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:06:16.913 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:53:51 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:06:17.061 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:06:17.062 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:06:17.063 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 99 M 2024-03-25T17:06:17.063 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 449 M 2024-03-25T17:06:17.063 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:08:44.394 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 402 kB/s | 287 MB 12:11 2024-03-25T17:08:44.395 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:44.395 INFO:teuthology.orchestra.run.smithi044.stdout:Total 402 kB/s | 287 MB 12:11 2024-03-25T17:08:44.395 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:08:44.398 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:08:44.398 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:08:46.189 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:08:46.189 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:09:17.313 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:09:17.425 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:09:18.650 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:09:18.923 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:09:18.923 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:18.923 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:09:18.924 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:09:18.924 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:18.924 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:09:19.139 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:09:19.718 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:56:51 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:09:19.873 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:09:19.874 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:19.874 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:09:19.874 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:19.874 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:09:19.874 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:09:19.874 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:19.874 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:09:19.875 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:19.875 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:09:19.875 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:19.875 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 5.9 M 2024-03-25T17:09:19.875 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 28 M 2024-03-25T17:09:19.875 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:09:35.750 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 383 kB/s | 5.9 MB 00:15 2024-03-25T17:09:35.751 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:09:35.751 INFO:teuthology.orchestra.run.smithi044.stdout:Total 383 kB/s | 5.9 MB 00:15 2024-03-25T17:09:35.751 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:09:35.756 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:09:35.756 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:09:35.833 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:09:35.833 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:09:36.602 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:09:36.710 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:09:37.302 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:09:37.571 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:09:37.571 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:37.571 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:09:37.571 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:09:37.571 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:37.571 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:09:37.760 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:09:38.335 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:57:10 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:09:38.493 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:38.495 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:09:38.495 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:38.495 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 109 k 2024-03-25T17:09:38.495 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 440 k 2024-03-25T17:09:38.495 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:09:39.404 INFO:teuthology.orchestra.run.smithi044.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 120 kB/s | 109 kB 00:00 2024-03-25T17:09:39.404 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:09:39.404 INFO:teuthology.orchestra.run.smithi044.stdout:Total 119 kB/s | 109 kB 00:00 2024-03-25T17:09:39.405 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:09:39.409 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:09:39.409 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:09:39.447 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:09:39.448 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:09:39.572 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:09:39.689 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:09:40.077 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:09:40.332 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:09:40.333 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:40.333 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:09:40.333 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:09:40.333 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:40.333 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:09:40.513 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:09:41.093 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:57:13 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:09:41.248 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:09:41.249 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 5.4 M 2024-03-25T17:09:41.250 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 25 M 2024-03-25T17:09:41.250 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:10:04.871 INFO:teuthology.orchestra.run.smithi044.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 232 kB/s | 5.4 MB 00:23 2024-03-25T17:10:04.871 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:10:04.872 INFO:teuthology.orchestra.run.smithi044.stdout:Total 232 kB/s | 5.4 MB 00:23 2024-03-25T17:10:04.872 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:10:04.876 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:10:04.877 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:10:04.948 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:10:04.948 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:10:05.645 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:10:05.873 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:10:06.391 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:10:06.786 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:10:06.786 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:10:06.786 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:10:06.786 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:06.786 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:10:06.786 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:10:06.976 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados2-debuginfo 2024-03-25T17:10:07.553 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:57:39 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:10:07.707 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:10:07.708 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 44 M 2024-03-25T17:10:07.709 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 253 M 2024-03-25T17:10:07.709 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:10:46.024 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 375 kB/s | 99 MB 04:28 2024-03-25T17:10:46.025 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:10:46.025 INFO:teuthology.orchestra.run.smithi111.stdout:Total 375 kB/s | 99 MB 04:28 2024-03-25T17:10:46.026 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:10:46.029 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:10:46.029 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:10:46.637 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:10:46.637 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:10:56.913 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:10:57.012 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:10:59.954 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:11:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:11:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:11:00.217 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:11:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:11:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:11:00.218 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:11:00.414 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:11:00.979 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:58:35 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:11:01.127 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:11:01.128 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:11:01.129 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 287 M 2024-03-25T17:11:01.129 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.2 G 2024-03-25T17:11:01.129 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:13:26.953 INFO:teuthology.orchestra.run.smithi044.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 228 kB/s | 44 MB 03:19 2024-03-25T17:13:26.954 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:26.954 INFO:teuthology.orchestra.run.smithi044.stdout:Total 228 kB/s | 44 MB 03:19 2024-03-25T17:13:26.954 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:13:26.957 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:13:26.957 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:13:27.270 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:13:27.270 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:13:32.605 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:13:32.696 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:13:34.590 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:13:34.863 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:13:34.863 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:13:34.863 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:13:34.863 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:34.863 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:13:34.863 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:13:35.043 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:13:35.617 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:01:07 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:13:35.773 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:13:35.774 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:13:35.774 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:13:35.774 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:13:35.774 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:13:35.774 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:13:35.774 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:13:35.774 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:13:35.775 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:13:35.775 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:13:35.775 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:13:35.775 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 52 M 2024-03-25T17:13:35.775 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 308 M 2024-03-25T17:13:35.775 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:15:16.720 INFO:teuthology.orchestra.run.smithi044.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 526 kB/s | 52 MB 01:40 2024-03-25T17:15:16.721 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:16.721 INFO:teuthology.orchestra.run.smithi044.stdout:Total 526 kB/s | 52 MB 01:40 2024-03-25T17:15:16.721 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:15:16.724 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:15:16.725 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:15:17.083 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:15:17.084 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:15:23.423 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:15:23.521 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:25.655 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:25.912 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:25.912 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:25.912 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:15:25.912 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:25.912 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:25.912 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:15:26.091 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:15:26.657 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:02:58 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:15:26.813 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:15:26.813 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 615 k 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.1 M 2024-03-25T17:15:26.814 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:15:29.622 INFO:teuthology.orchestra.run.smithi044.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 219 kB/s | 615 kB 00:02 2024-03-25T17:15:29.623 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:29.623 INFO:teuthology.orchestra.run.smithi044.stdout:Total 219 kB/s | 615 kB 00:02 2024-03-25T17:15:29.623 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:15:29.627 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:15:29.628 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:15:29.671 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:15:29.671 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:15:29.846 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:15:29.948 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:15:30.399 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:15:30.659 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:15:30.659 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:30.659 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:15:30.659 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:30.659 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:30.659 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:15:30.827 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:15:31.402 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:03:03 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:15:31.558 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:15:31.559 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:31.559 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:15:31.559 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:31.559 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:15:31.559 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:15:31.559 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:15:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:15:31.560 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 771 k 2024-03-25T17:15:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.7 M 2024-03-25T17:15:31.560 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:15:35.108 INFO:teuthology.orchestra.run.smithi044.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 217 kB/s | 771 kB 00:03 2024-03-25T17:15:35.109 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:35.109 INFO:teuthology.orchestra.run.smithi044.stdout:Total 217 kB/s | 771 kB 00:03 2024-03-25T17:15:35.109 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:15:35.113 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:15:35.114 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:15:35.157 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:15:35.158 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:15:35.411 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:15:35.519 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:15:35.894 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:15:36.162 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:15:36.163 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:36.163 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:15:36.163 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:36.163 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:36.163 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:15:36.375 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:15:36.947 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:03:08 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:15:37.102 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:15:37.102 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:37.102 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 1.1 M 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 4.3 M 2024-03-25T17:15:37.103 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:15:42.328 INFO:teuthology.orchestra.run.smithi044.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 220 kB/s | 1.1 MB 00:05 2024-03-25T17:15:42.328 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:42.329 INFO:teuthology.orchestra.run.smithi044.stdout:Total 220 kB/s | 1.1 MB 00:05 2024-03-25T17:15:42.329 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:15:42.333 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:15:42.333 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:15:42.378 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:15:42.378 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:15:42.610 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:15:42.721 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:43.117 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:43.376 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:43.377 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:15:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:43.377 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:43.377 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:15:43.575 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:15:44.150 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:03:16 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:15:44.306 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:15:44.307 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:44.307 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:15:44.307 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:44.307 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:15:44.307 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:15:44.307 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:44.307 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:15:44.308 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:44.308 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:15:44.308 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:44.308 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 337 k 2024-03-25T17:15:44.308 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 1.0 M 2024-03-25T17:15:44.308 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:15:45.655 INFO:teuthology.orchestra.run.smithi044.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 250 kB/s | 337 kB 00:01 2024-03-25T17:15:45.656 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:45.656 INFO:teuthology.orchestra.run.smithi044.stdout:Total 250 kB/s | 337 kB 00:01 2024-03-25T17:15:45.657 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:15:45.660 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:15:45.661 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:15:45.702 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:15:45.703 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:15:45.848 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:15:45.974 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:46.342 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:46.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:46.603 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:46.603 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:15:46.603 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:46.603 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:46.603 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:15:46.798 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:15:47.376 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:03:19 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:15:47.532 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 509 k 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.5 M 2024-03-25T17:15:47.533 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:15:50.591 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 166 kB/s | 509 kB 00:03 2024-03-25T17:15:50.592 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:50.592 INFO:teuthology.orchestra.run.smithi044.stdout:Total 166 kB/s | 509 kB 00:03 2024-03-25T17:15:50.593 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:15:50.597 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:15:50.597 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:15:50.638 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:15:50.638 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:15:50.806 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:15:50.910 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:15:51.315 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:15:51.568 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:15:51.568 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:51.568 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:15:51.568 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:51.569 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:51.569 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:15:51.789 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:15:52.362 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:03:24 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:15:52.518 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:15:52.518 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:15:52.519 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:15:52.520 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 52 M 2024-03-25T17:15:52.520 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 298 M 2024-03-25T17:15:52.520 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:18:47.809 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 301 kB/s | 52 MB 02:55 2024-03-25T17:18:47.840 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:18:47.840 INFO:teuthology.orchestra.run.smithi044.stdout:Total 301 kB/s | 52 MB 02:55 2024-03-25T17:18:47.840 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:18:47.840 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:18:47.841 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:18:48.164 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:18:48.164 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:18:54.410 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:18:54.510 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:18:56.518 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:18:56.778 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:18:56.778 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:18:56.778 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:18:56.778 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:18:56.778 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:18:56.778 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:18:56.972 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:18:57.554 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:06:29 ago on Mon 25 Mar 2024 04:12:28 PM UTC. 2024-03-25T17:18:57.709 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T17:18:57.710 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:18:57.710 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T17:18:57.710 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 1.3 M 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 5.5 M 2024-03-25T17:18:57.711 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T17:19:02.611 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 262 kB/s | 1.3 MB 00:04 2024-03-25T17:19:02.612 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:19:02.612 INFO:teuthology.orchestra.run.smithi044.stdout:Total 262 kB/s | 1.3 MB 00:04 2024-03-25T17:19:02.612 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T17:19:02.616 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T17:19:02.616 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T17:19:02.662 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T17:19:02.662 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T17:19:02.928 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T17:19:03.038 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:19:03.445 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:19:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:19:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:19:03.728 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T17:19:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:19:03.728 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:19:03.728 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T17:19:03.905 DEBUG:teuthology.parallel:result is None 2024-03-25T17:27:37.541 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 295 kB/s | 287 MB 16:36 2024-03-25T17:27:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:27:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:Total 295 kB/s | 287 MB 16:36 2024-03-25T17:27:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:27:37.546 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:27:37.546 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:27:39.249 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:27:39.249 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:28:08.413 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:28:08.511 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:28:16.034 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:28:16.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:28:16.305 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:16.305 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:28:16.305 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:28:16.305 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:16.305 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:28:16.524 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:28:17.094 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:15:52 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:28:17.241 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:17.242 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:28:17.243 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:17.243 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 5.9 M 2024-03-25T17:28:17.243 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 28 M 2024-03-25T17:28:17.243 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:28:35.971 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 325 kB/s | 5.9 MB 00:18 2024-03-25T17:28:35.972 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:35.972 INFO:teuthology.orchestra.run.smithi111.stdout:Total 325 kB/s | 5.9 MB 00:18 2024-03-25T17:28:35.972 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:28:35.976 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:28:35.976 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:28:36.049 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:28:36.050 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:28:36.785 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:28:36.894 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:28:37.491 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:28:37.761 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:28:37.761 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:37.761 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:28:37.761 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:28:37.761 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:37.761 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:28:37.951 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:28:38.508 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:16:13 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:28:38.658 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:28:38.658 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 109 k 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 440 k 2024-03-25T17:28:38.659 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:28:39.615 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 114 kB/s | 109 kB 00:00 2024-03-25T17:28:39.616 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:39.616 INFO:teuthology.orchestra.run.smithi111.stdout:Total 114 kB/s | 109 kB 00:00 2024-03-25T17:28:39.616 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:28:39.620 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:28:39.620 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:28:39.655 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:28:39.656 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:28:39.771 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:28:39.880 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:28:40.268 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:28:40.543 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:28:40.543 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:40.544 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:28:40.544 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:28:40.544 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:40.544 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:28:40.740 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:28:41.302 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:16:16 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:28:41.451 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:28:41.452 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:41.452 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:28:41.452 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:41.452 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:28:41.452 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:28:41.453 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:41.453 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:28:41.453 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:41.453 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:28:41.453 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:41.453 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 5.4 M 2024-03-25T17:28:41.453 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 25 M 2024-03-25T17:28:41.453 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:28:53.708 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 447 kB/s | 5.4 MB 00:12 2024-03-25T17:28:53.709 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:53.710 INFO:teuthology.orchestra.run.smithi111.stdout:Total 447 kB/s | 5.4 MB 00:12 2024-03-25T17:28:53.710 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:28:53.714 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:28:53.714 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:28:53.783 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:28:53.783 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:28:54.450 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:28:54.566 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:28:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:28:55.391 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:28:55.392 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:55.392 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:28:55.392 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:28:55.392 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:55.392 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:28:55.601 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2-debuginfo 2024-03-25T17:28:56.159 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:16:31 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:28:56.307 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:28:56.308 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:28:56.309 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:28:56.309 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 44 M 2024-03-25T17:28:56.309 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 253 M 2024-03-25T17:28:56.309 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:31:55.238 INFO:teuthology.orchestra.run.smithi111.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 253 kB/s | 44 MB 02:58 2024-03-25T17:31:55.238 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:55.238 INFO:teuthology.orchestra.run.smithi111.stdout:Total 253 kB/s | 44 MB 02:58 2024-03-25T17:31:55.239 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:31:55.242 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:31:55.242 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:31:55.541 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:31:55.542 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:32:00.588 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:32:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:02.595 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:02.861 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:02.861 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:32:02.861 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:32:02.861 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:32:02.862 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:32:02.862 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:32:03.078 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:32:03.645 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:19:38 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:32:03.794 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:32:03.794 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:32:03.794 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:32:03.794 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:32:03.795 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:32:03.795 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:32:03.795 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:32:03.795 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:32:03.795 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:32:03.795 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:32:03.795 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:32:03.795 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 52 M 2024-03-25T17:32:03.796 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 308 M 2024-03-25T17:32:03.796 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:34:49.577 INFO:teuthology.orchestra.run.smithi111.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 320 kB/s | 52 MB 02:45 2024-03-25T17:34:49.578 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:34:49.578 INFO:teuthology.orchestra.run.smithi111.stdout:Total 320 kB/s | 52 MB 02:45 2024-03-25T17:34:49.578 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:34:49.581 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:34:49.581 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:34:49.926 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:34:49.926 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:34:55.909 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:34:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:34:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:34:58.609 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:34:58.609 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:34:58.610 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:34:58.610 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:34:58.610 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:34:58.610 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:34:58.798 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:34:59.360 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:22:34 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:34:59.506 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:34:59.508 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:34:59.509 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 615 k 2024-03-25T17:34:59.509 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.1 M 2024-03-25T17:34:59.509 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:35:01.795 INFO:teuthology.orchestra.run.smithi111.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 269 kB/s | 615 kB 00:02 2024-03-25T17:35:01.796 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:01.796 INFO:teuthology.orchestra.run.smithi111.stdout:Total 269 kB/s | 615 kB 00:02 2024-03-25T17:35:01.797 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:35:01.801 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:35:01.801 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:35:01.842 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:35:01.843 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:35:02.013 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:35:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:35:02.540 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:35:02.802 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:35:02.802 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:02.802 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:35:02.802 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:35:02.802 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:02.802 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:35:03.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:35:03.568 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:22:38 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:35:03.717 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 771 k 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.7 M 2024-03-25T17:35:03.718 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:35:06.573 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 270 kB/s | 771 kB 00:02 2024-03-25T17:35:06.574 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:06.574 INFO:teuthology.orchestra.run.smithi111.stdout:Total 270 kB/s | 771 kB 00:02 2024-03-25T17:35:06.575 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:35:06.578 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:35:06.579 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:35:06.620 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:35:06.620 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:35:06.801 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:35:06.905 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:35:07.300 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:35:07.583 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:35:07.583 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:07.584 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:35:07.584 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:35:07.584 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:07.584 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:35:07.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:35:08.387 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:22:43 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:35:08.537 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:35:08.538 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:08.538 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.1 M 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 4.3 M 2024-03-25T17:35:08.539 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:35:13.401 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 237 kB/s | 1.1 MB 00:04 2024-03-25T17:35:13.402 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:13.402 INFO:teuthology.orchestra.run.smithi111.stdout:Total 237 kB/s | 1.1 MB 00:04 2024-03-25T17:35:13.402 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:35:13.406 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:35:13.406 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:35:13.447 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:35:13.448 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:35:13.662 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:35:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:35:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:35:14.482 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:35:14.482 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:14.482 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:35:14.482 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:35:14.482 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:14.482 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:35:14.699 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:35:15.266 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:22:50 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:35:15.416 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:35:15.417 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:35:15.417 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:15.417 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:35:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:35:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:15.418 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:35:15.418 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:15.418 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:35:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:15.418 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 337 k 2024-03-25T17:35:15.418 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.0 M 2024-03-25T17:35:15.418 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:35:16.596 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 286 kB/s | 337 kB 00:01 2024-03-25T17:35:16.596 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:16.597 INFO:teuthology.orchestra.run.smithi111.stdout:Total 286 kB/s | 337 kB 00:01 2024-03-25T17:35:16.597 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:35:16.601 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:35:16.601 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:35:16.641 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:35:16.642 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:35:16.784 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:35:16.886 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:35:17.319 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:35:17.590 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:35:17.590 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:17.590 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:35:17.590 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:35:17.591 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:17.591 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:35:17.796 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:35:18.355 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:22:53 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:35:18.505 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 509 k 2024-03-25T17:35:18.506 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.5 M 2024-03-25T17:35:18.507 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:35:20.340 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 278 kB/s | 509 kB 00:01 2024-03-25T17:35:20.341 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:20.342 INFO:teuthology.orchestra.run.smithi111.stdout:Total 277 kB/s | 509 kB 00:01 2024-03-25T17:35:20.342 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:35:20.346 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:35:20.346 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:35:20.385 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:35:20.385 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:35:20.548 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:35:20.647 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:35:21.069 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:35:21.331 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:35:21.331 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:21.331 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:35:21.331 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:35:21.331 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:21.331 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:35:21.528 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:35:22.092 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:22:57 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:35:22.241 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:35:22.242 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:22.242 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:35:22.242 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:22.242 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:35:22.242 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:35:22.242 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:22.242 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:35:22.243 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:35:22.243 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:35:22.243 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:35:22.243 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 52 M 2024-03-25T17:35:22.243 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 298 M 2024-03-25T17:35:22.243 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:37:52.757 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 351 kB/s | 52 MB 02:30 2024-03-25T17:37:52.757 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:52.758 INFO:teuthology.orchestra.run.smithi111.stdout:Total 351 kB/s | 52 MB 02:30 2024-03-25T17:37:52.758 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:37:52.761 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:37:52.761 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:37:53.095 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:37:53.095 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:37:59.000 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:37:59.105 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:38:01.157 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:38:01.422 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:38:01.422 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:01.422 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:38:01.423 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:01.423 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:01.423 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:38:01.616 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:38:02.186 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:25:37 ago on Mon 25 Mar 2024 04:12:25 PM UTC. 2024-03-25T17:38:02.335 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T17:38:02.336 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:38:02.336 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.3 M 2024-03-25T17:38:02.337 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 5.5 M 2024-03-25T17:38:02.338 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T17:38:05.175 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 452 kB/s | 1.3 MB 00:02 2024-03-25T17:38:05.176 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:05.176 INFO:teuthology.orchestra.run.smithi111.stdout:Total 452 kB/s | 1.3 MB 00:02 2024-03-25T17:38:05.176 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T17:38:05.180 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T17:38:05.180 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T17:38:05.223 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T17:38:05.224 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T17:38:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T17:38:05.598 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:38:06.000 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:38:06.272 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:38:06.272 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:06.272 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T17:38:06.272 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:06.272 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:06.272 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T17:38:06.486 DEBUG:teuthology.parallel:result is None 2024-03-25T17:38:06.487 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:38:06.711 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T17:38:06.735 INFO:teuthology.orchestra.run.smithi044.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T17:38:06.736 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T17:38:06.736 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T17:38:06.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:38:06.879 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T17:38:06.899 INFO:teuthology.orchestra.run.smithi111.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T17:38:06.900 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T17:38:06.900 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T17:38:06.901 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T17:38:06.901 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:06.901 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T17:38:06.933 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:06.933 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T17:38:06.967 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T17:38:06.968 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:06.968 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T17:38:07.003 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T17:38:07.072 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:07.072 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T17:38:07.099 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T17:38:07.165 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T17:38:07.165 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:07.166 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T17:38:07.195 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T17:38:07.264 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:07.264 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T17:38:07.290 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T17:38:07.356 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T17:38:07.357 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:07.357 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T17:38:07.386 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T17:38:07.455 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:07.455 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T17:38:07.482 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T17:38:07.548 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T17:38:07.625 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T17:38:07.625 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /var/log/ceph 2024-03-25T17:38:07.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /var/log/ceph 2024-03-25T17:38:07.656 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T17:38:07.656 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T17:38:07.698 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T17:38:07.726 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T17:38:07.726 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T17:38:07.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T17:38:07.840 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T17:38:07.840 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T17:38:07.840 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621238', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi044.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK5AQGusdjqA96uup2BCfl1loIxJfl6NhrqvGoVH2U8OcUsKi/skdMWIWIDHuAdniI9UIgaBxVWJZrsc7DcoN4s=', 'smithi111.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcLfbwlvh5E0m3Gu/sf3hW7GNawTRIu453B3YyRAHMbkxUusdsYvD1JLkJPyoQpP75IrNRy6bdn/EADUKCRZLQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T17:38:07.841 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T17:38:07.843 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T17:38:07.858 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T17:38:07.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T17:38:07.928 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:07.928 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T17:38:07.985 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T17:38:07.985 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-03-25 16:14:32.717498380 +0000 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-03-25 16:11:38.756237857 +0000 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-03-25 16:11:38.756237857 +0000 2024-03-25T17:38:08.042 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-03-25 16:11:38.756237857 +0000 2024-03-25T17:38:08.042 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T17:38:08.110 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-03-25T17:38:08.110 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-03-25T17:38:08.110 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000200735 s, 2.6 MB/s 2024-03-25T17:38:08.112 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T17:38:08.170 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2024-03-25T17:38:08.226 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T17:38:08.227 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:38:08.227 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T17:38:08.227 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:38:08.227 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:38:08.227 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-03-25 16:14:32.718498363 +0000 2024-03-25T17:38:08.227 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-03-25 16:11:39.102231888 +0000 2024-03-25T17:38:08.227 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-03-25 16:11:39.102231888 +0000 2024-03-25T17:38:08.227 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-03-25 16:11:39.102231888 +0000 2024-03-25T17:38:08.227 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T17:38:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-03-25T17:38:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-03-25T17:38:08.296 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000200904 s, 2.5 MB/s 2024-03-25T17:38:08.298 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T17:38:08.356 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2024-03-25T17:38:08.412 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T17:38:08.412 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:38:08.413 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T17:38:08.413 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:38:08.413 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:38:08.413 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-03-25 16:14:32.718498363 +0000 2024-03-25T17:38:08.413 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-03-25 16:11:39.442226022 +0000 2024-03-25T17:38:08.413 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-03-25 16:11:39.442226022 +0000 2024-03-25T17:38:08.413 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-03-25 16:11:39.442226022 +0000 2024-03-25T17:38:08.413 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T17:38:08.481 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-03-25T17:38:08.481 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-03-25T17:38:08.481 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.00022117 s, 2.3 MB/s 2024-03-25T17:38:08.482 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T17:38:08.539 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2024-03-25T17:38:08.595 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T17:38:08.596 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:38:08.596 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T17:38:08.596 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:38:08.596 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:38:08.596 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-03-25 16:14:32.718498363 +0000 2024-03-25T17:38:08.596 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-03-25 16:11:39.769220380 +0000 2024-03-25T17:38:08.596 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-03-25 16:11:39.769220380 +0000 2024-03-25T17:38:08.596 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-03-25 16:11:39.769220380 +0000 2024-03-25T17:38:08.596 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T17:38:08.664 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-03-25T17:38:08.664 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-03-25T17:38:08.664 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000202211 s, 2.5 MB/s 2024-03-25T17:38:08.665 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T17:38:08.722 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:08.722 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:08.722 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T17:38:08.741 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T17:38:08.741 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-03-25T17:38:08.798 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T17:38:08.798 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:38:08.799 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T17:38:08.799 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:38:08.799 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:38:08.799 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-25 16:14:26.147718425 +0000 2024-03-25T17:38:08.799 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-25 16:09:24.340323032 +0000 2024-03-25T17:38:08.799 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-25 16:09:24.340323032 +0000 2024-03-25T17:38:08.799 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-25 16:09:24.340323032 +0000 2024-03-25T17:38:08.799 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T17:38:08.868 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-25T17:38:08.868 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-25T17:38:08.868 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000224169 s, 2.3 MB/s 2024-03-25T17:38:08.869 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T17:38:08.928 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-03-25T17:38:08.984 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T17:38:08.984 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:38:08.984 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T17:38:08.984 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:38:08.985 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:38:08.985 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-25 16:14:26.147718425 +0000 2024-03-25T17:38:08.985 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-25 16:09:24.679316850 +0000 2024-03-25T17:38:08.985 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-25 16:09:24.679316850 +0000 2024-03-25T17:38:08.985 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-25 16:09:24.679316850 +0000 2024-03-25T17:38:08.985 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T17:38:09.051 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-25T17:38:09.052 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-25T17:38:09.052 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000171886 s, 3.0 MB/s 2024-03-25T17:38:09.053 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T17:38:09.109 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-25 16:14:26.148718407 +0000 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-25 16:09:25.031310431 +0000 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-25 16:09:25.031310431 +0000 2024-03-25T17:38:09.166 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-25 16:09:25.031310431 +0000 2024-03-25T17:38:09.167 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T17:38:09.233 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-25T17:38:09.233 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-25T17:38:09.233 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000178368 s, 2.9 MB/s 2024-03-25T17:38:09.234 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T17:38:09.290 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-03-25T17:38:09.345 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T17:38:09.345 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:38:09.345 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T17:38:09.345 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:38:09.345 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:38:09.345 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-25 16:14:26.148718407 +0000 2024-03-25T17:38:09.345 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-25 16:09:25.375304158 +0000 2024-03-25T17:38:09.346 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-25 16:09:25.375304158 +0000 2024-03-25T17:38:09.346 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-25 16:09:25.375304158 +0000 2024-03-25T17:38:09.346 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T17:38:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-25T17:38:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-25T17:38:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000159149 s, 3.2 MB/s 2024-03-25T17:38:09.412 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T17:38:09.467 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:09.468 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi044.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T17:38:09.468 INFO:tasks.ceph:Generating config... 2024-03-25T17:38:09.469 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T17:38:09.469 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T17:38:09.469 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T17:38:09.469 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T17:38:09.469 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T17:38:09.469 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T17:38:09.469 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T17:38:09.469 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T17:38:09.469 INFO:tasks.ceph:[global] ms type = async 2024-03-25T17:38:09.470 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T17:38:09.470 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T17:38:09.470 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T17:38:09.471 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T17:38:09.472 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T17:38:09.472 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T17:38:09.473 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T17:38:09.531 INFO:teuthology.orchestra.run.smithi044.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T17:38:09.534 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T17:38:09.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:38:09.608 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.44:6789]'), ('mon.c', '[v1:172.21.15.44:6790]'), ('mon.b', '[v1:172.21.15.111:6789]')] 2024-03-25T17:38:09.609 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.44:6789],[v1:172.21.15.44:6790],[v1:172.21.15.111:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T17:38:09.609 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:09.609 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T17:38:09.663 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.44:6789]' --addv c '[v1:172.21.15.44:6790]' --addv b '[v1:172.21.15.111:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: generated fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:setting min_mon_release = quincy 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 0 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:min_mon_release 17 (quincy) 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:election_strategy: 3 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:38:09.749 INFO:teuthology.orchestra.run.smithi044.stdout:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:38:09.750 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T17:38:09.751 DEBUG:teuthology.orchestra.run.smithi044:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T17:38:09.765 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6a379495-4f08-496c-b095-a059f345ad04... 2024-03-25T17:38:09.766 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T17:38:09.808 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T17:38:09.855 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-03-25T17:38:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: chdir = "" 2024-03-25T17:38:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T17:38:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: auth supported = cephx 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: filestore xattr use omap = true 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: mon clock drift allowed = 1.000 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: osd crush chooseleaf type = 0 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: auth debug = true 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on old message = true 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on bug = true 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: mon pg warn max object skew = 0 2024-03-25T17:38:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default size = 2 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow primary affinity = true 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow pg remap = true 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on legacy crush tunables = false 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on crush straw calc version zero = false 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on no sortbitwise = false 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on osd down out interval zero = false 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on too few osds = false 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T17:38:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: mon_allow_pool_size_one = true 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: osd default data pool replay window = 5 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: mon allow pool delete = true 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: mon cluster log file level = debug 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: debug asserts on shutdown = true 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: mon health detail to clog = false 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: mon host = "[v1:172.21.15.44:6789],[v1:172.21.15.44:6790],[v1:172.21.15.111:6789]" 2024-03-25T17:38:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: mon client directed command retry = 5 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: mon election default strategy = 3 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: ms bind msgr2 = False 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: ms inject socket failures = 5000 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: ms type = async 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: osd heartbeat grace = 80 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout:[osd] 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: osd journal size = 100 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub load threshold = 5.0 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub max interval = 600 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock profile = high_recovery_ops 2024-03-25T17:38:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd recover clone overlap = true 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd recovery max chunk = 1048576 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug shutdown = true 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug op order = true 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify stray on activate = true 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug trim objects = true 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd open classes on start = true 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug pg log writeout = true 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd deep scrub update digest min age = 30 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: osd map max advance = 10 2024-03-25T17:38:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: journal zero on create = true 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: filestore ondisk finisher threads = 3 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: filestore apply finisher threads = 3 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: bdev debug aio = true 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug misdirected ops = true 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: bdev async discard = True 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: bdev enable discard = True 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore allocator = stupid 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore block size = 96636764160 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore fsck on mount = True 2024-03-25T17:38:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore zero block detection = True 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluefs = 1 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluestore = 1 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: debug monc = 20 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: debug osd = 20 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: debug rocksdb = 10 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd full ratio = 0.9 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd nearfull ratio = 0.8 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: osd blocked scrub grace period = 3600 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: osd class default list = * 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: osd class load list = * 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: osd client watch timeout = 120 2024-03-25T17:38:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify cached snaps = True 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify missing on start = True 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd failsafe full ratio = 0.95 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd fast shutdown = False 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd max markdown count = 1000 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock override recovery settings = True 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd objectstore = bluestore 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd op queue = debug_random 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd op queue cut off = debug_random 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: osd shutdown pgref assert = True 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr] 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: debug mgr = 20 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-03-25T17:38:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: mgr/telemetry/nag = false 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout:[mon] 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: debug paxos = 20 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: mon data avail warn = 5 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: mon mgr mkfs grace = 240 2024-03-25T17:38:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd reporter subtree level = osd 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd prime pg temp = true 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T17:38:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: # 1m isn't quite enough 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon_down_mkfs_grace = 2m 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_filestore_osds = false 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon min osdmap epochs = 50 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd crush smoke test = False 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon osdmap full prune interval = 2 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon osdmap full prune min = 15 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon osdmap full prune txsize = 2 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon scrub interval = 300 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on pool no app = False 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: paxos service trim min = 10 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:38:09.866 INFO:teuthology.orchestra.run.smithi044.stdout:[client] 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: rgw cache enabled = true 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable ops log = true 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable usage log = true 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: debug monc = 20 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: debug objecter = 20 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: debug rados = 20 2024-03-25T17:38:09.867 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.a] 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.c] 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.b] 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.868 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2024-03-25T17:38:09.869 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T17:38:09.870 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "[v1:172.21.15.44:6789],[v1:172.21.15.44:6790],[v1:172.21.15.111:6789]" 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: mon client directed command retry = 5 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: mon election default strategy = 3 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: ms bind msgr2 = False 2024-03-25T17:38:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: ms inject socket failures = 5000 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: ms type = async 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: osd heartbeat grace = 80 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock profile = high_recovery_ops 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2024-03-25T17:38:09.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug trim objects = true 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2024-03-25T17:38:09.873 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: bdev async discard = True 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: bdev enable discard = True 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore allocator = stupid 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore zero block detection = True 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 1 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 1 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: debug monc = 20 2024-03-25T17:38:09.874 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 10 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd blocked scrub grace period = 3600 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd class default list = * 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd class load list = * 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd client watch timeout = 120 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify cached snaps = True 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify missing on start = True 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd fast shutdown = False 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd max markdown count = 1000 2024-03-25T17:38:09.875 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock override recovery settings = True 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: osd op queue = debug_random 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: osd op queue cut off = debug_random 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: osd shutdown pgref assert = True 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2024-03-25T17:38:09.876 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: mon min osdmap epochs = 50 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd crush smoke test = False 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: mon osdmap full prune interval = 2 2024-03-25T17:38:09.878 INFO:teuthology.orchestra.run.smithi111.stdout: mon osdmap full prune min = 15 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: mon osdmap full prune txsize = 2 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: mon scrub interval = 300 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on pool no app = False 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: paxos service trim min = 10 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: debug monc = 20 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: debug objecter = 20 2024-03-25T17:38:09.879 INFO:teuthology.orchestra.run.smithi111.stdout: debug rados = 20 2024-03-25T17:38:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2024-03-25T17:38:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2024-03-25T17:38:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2024-03-25T17:38:09.881 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T17:38:09.881 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T17:38:09.936 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T17:38:09.936 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:09.936 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T17:38:09.950 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:09.951 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T17:38:10.005 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:38:10.005 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:10.005 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T17:38:10.006 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:38:10.084 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:10.085 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:10.138 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:38:10.138 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:10.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T17:38:10.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:38:10.178 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:10.178 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:10.232 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T17:38:10.233 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T17:38:10.233 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T17:38:10.287 INFO:teuthology.orchestra.run.smithi044.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T17:38:10.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T17:38:10.362 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T17:38:10.364 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T17:38:10.364 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T17:38:10.365 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T17:38:10.406 INFO:teuthology.orchestra.run.smithi044.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T17:38:10.421 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T17:38:10.463 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T17:38:10.478 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T17:38:10.478 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi044.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T17:38:10.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T17:38:10.505 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:10.505 INFO:tasks.ceph:role: osd.0 2024-03-25T17:38:10.505 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:38:10.505 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T17:38:10.578 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:38:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:38:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:38:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:38:10.579 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:38:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:38:10.579 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:38:10.579 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:38:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:38:10.579 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:38:11.086 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-03-25T17:38:11.088 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-03-25T17:38:11.088 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T17:38:11.220 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T17:38:11.253 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T17:38:11.320 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:11.320 INFO:tasks.ceph:role: osd.1 2024-03-25T17:38:11.320 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:38:11.320 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T17:38:11.394 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:38:11.395 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:38:11.847 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-03-25T17:38:11.849 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-03-25T17:38:11.849 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T17:38:11.942 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T17:38:11.973 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T17:38:12.041 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:12.041 INFO:tasks.ceph:role: osd.2 2024-03-25T17:38:12.041 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:38:12.041 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T17:38:12.115 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:38:12.115 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:38:12.115 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:38:12.115 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:38:12.115 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:38:12.115 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:38:12.116 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:38:12.116 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:38:12.116 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:38:12.116 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:38:12.600 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-03-25T17:38:12.602 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-03-25T17:38:12.602 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T17:38:12.752 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T17:38:12.785 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T17:38:12.851 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:12.851 INFO:tasks.ceph:role: osd.3 2024-03-25T17:38:12.851 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:38:12.851 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T17:38:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:38:12.925 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:38:12.925 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:38:12.925 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:38:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:38:12.925 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:38:12.925 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:38:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:38:12.926 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:38:12.926 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:38:13.424 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-03-25T17:38:13.427 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-03-25T17:38:13.427 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T17:38:13.563 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T17:38:13.597 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:13.696+0000 7f0849cb5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T17:38:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:13.697+0000 7f0849cb5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T17:38:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:13.697+0000 7f0849cb5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:38:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:13.697+0000 7f0849cb5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:38:13.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:13.697+0000 7f0849cb5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:38:13.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:13.697+0000 7f0849cb5640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T17:38:16.539 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T17:38:16.570 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:16.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:16.661+0000 7f8fd7381640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T17:38:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:16.662+0000 7f8fd7381640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T17:38:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:16.662+0000 7f8fd7381640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:38:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:16.662+0000 7f8fd7381640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:38:16.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:16.662+0000 7f8fd7381640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:38:16.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:16.662+0000 7f8fd7381640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T17:38:19.512 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T17:38:19.544 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:19.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:19.634+0000 7fdd72d38640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T17:38:19.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:19.635+0000 7fdd72d38640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T17:38:19.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:19.635+0000 7fdd72d38640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:38:19.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:19.635+0000 7fdd72d38640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:38:19.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:19.635+0000 7fdd72d38640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:38:19.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:19.635+0000 7fdd72d38640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T17:38:22.481 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T17:38:22.512 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:22.602+0000 7fd17e613640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T17:38:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:22.603+0000 7fd17e613640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T17:38:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:22.603+0000 7fd17e613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:38:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:22.603+0000 7fd17e613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:38:22.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:22.603+0000 7fd17e613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:38:22.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:22.603+0000 7fd17e613640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T17:38:25.446 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T17:38:25.477 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T17:38:25.509 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:25.509 INFO:tasks.ceph:role: osd.4 2024-03-25T17:38:25.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:38:25.510 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T17:38:25.587 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:38:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:38:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:38:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:38:25.587 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:38:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:38:25.587 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:38:25.588 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:38:25.588 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:38:25.588 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:38:26.085 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-03-25T17:38:26.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-03-25T17:38:26.087 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T17:38:26.238 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T17:38:26.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T17:38:26.342 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:26.342 INFO:tasks.ceph:role: osd.5 2024-03-25T17:38:26.342 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:38:26.342 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T17:38:26.418 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:38:26.418 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:38:26.418 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:38:26.418 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:38:26.418 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:38:26.418 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:38:26.418 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:38:26.419 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:38:26.419 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:38:26.419 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:38:26.867 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-03-25T17:38:26.870 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-03-25T17:38:26.870 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T17:38:26.973 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T17:38:27.007 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T17:38:27.075 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:27.076 INFO:tasks.ceph:role: osd.6 2024-03-25T17:38:27.076 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:38:27.076 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:38:27.152 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:38:27.643 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-03-25T17:38:27.646 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-03-25T17:38:27.646 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T17:38:27.807 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T17:38:27.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T17:38:27.910 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:38:27.910 INFO:tasks.ceph:role: osd.7 2024-03-25T17:38:27.910 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:38:27.910 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T17:38:27.985 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:38:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:38:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:38:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:38:27.986 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:38:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:38:27.986 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:38:27.986 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:38:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:38:27.986 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:38:28.498 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-03-25T17:38:28.500 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-03-25T17:38:28.500 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T17:38:28.613 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T17:38:28.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:28.756+0000 7f93f7a37640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T17:38:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:28.756+0000 7f93f7a37640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T17:38:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:28.756+0000 7f93f7a37640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:38:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:28.756+0000 7f93f7a37640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:38:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:28.756+0000 7f93f7a37640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:38:28.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:28.757+0000 7f93f7a37640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T17:38:31.600 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T17:38:31.629 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:31.718+0000 7f0ed4c0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T17:38:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:31.719+0000 7f0ed4c0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T17:38:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:31.719+0000 7f0ed4c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:38:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:31.719+0000 7f0ed4c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:38:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:31.719+0000 7f0ed4c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:38:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:31.719+0000 7f0ed4c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T17:38:34.582 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T17:38:34.611 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:34.701+0000 7f83e84c8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T17:38:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:34.701+0000 7f83e84c8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T17:38:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:34.701+0000 7f83e84c8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:38:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:34.701+0000 7f83e84c8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:38:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:34.701+0000 7f83e84c8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:38:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:34.702+0000 7f83e84c8640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T17:38:37.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T17:38:37.578 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:37.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:37.669+0000 7f5891415640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T17:38:37.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:37.670+0000 7f5891415640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T17:38:37.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:37.670+0000 7f5891415640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:38:37.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:37.670+0000 7f5891415640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:38:37.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:37.670+0000 7f5891415640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:38:37.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:38:37.670+0000 7f5891415640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T17:38:40.519 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T17:38:40.551 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T17:38:40.551 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:40.551 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T17:38:40.582 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:40.582 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T17:38:40.652 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:40.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T17:38:40.722 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:40.722 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T17:38:40.791 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:40.791 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T17:38:40.859 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:40.859 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T17:38:40.890 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:40.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T17:38:40.960 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:40.961 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T17:38:41.031 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:41.032 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T17:38:41.101 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:41.101 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T17:38:41.169 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:38:41.170 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T17:38:41.185 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:38:41.186 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T17:38:41.226 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T17:38:41.226 DEBUG:teuthology.orchestra.run.smithi044:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T17:38:41.228 DEBUG:teuthology.orchestra.run.smithi111:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T17:38:41.269 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr.y] 2024-03-25T17:38:41.269 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCCtgFmrxssERAAAUMIEBATCvsznnA9DC/gZQ== 2024-03-25T17:38:41.269 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.0] 2024-03-25T17:38:41.269 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCFtgFm5CCYKRAABG2iIheR2vP4uw99gUa/bQ== 2024-03-25T17:38:41.269 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.1] 2024-03-25T17:38:41.269 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCItgFmMkGAJxAAC4OklsJ9bzbKo0N90xApyQ== 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.2] 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCLtgFmHSP2JRAACe2FmPWFa7JzdonNjKYh6w== 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.3] 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCOtgFm1+sMJBAAzl5auI/6NhouFoC2b9SG5g== 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr.x] 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCCtgFmrMKkFRAAbo8jKtFye/tlPv5Kg5YDOw== 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.4] 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCUtgFmXoIkLRAAfJVvghtqiUXpNBWHfuttcQ== 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.5] 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCXtgFmypXlKhAASaHeWJFFqi1DZDQNhYA3IA== 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.6] 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCatgFmjojgKRAAV2pgTQmFwO1zKoM2eBc7sQ== 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.7] 2024-03-25T17:38:41.270 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCdtgFm7U/8JxAAAj4JIOFa+n1/QPnY9w6O7A== 2024-03-25T17:38:41.271 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-03-25T17:38:41.271 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCCtgFmr/xGGBAAZgI5i5qYHZQfI+i0jZY5eg== 2024-03-25T17:38:41.271 INFO:teuthology.orchestra.run.smithi044.stdout:[client.1] 2024-03-25T17:38:41.271 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQCCtgFmguyuGxAAqG7F78yYrdxf9q3UMxnVQg== 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.y] 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCCtgFmrxssERAAAUMIEBATCvsznnA9DC/gZQ== 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.0] 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCFtgFm5CCYKRAABG2iIheR2vP4uw99gUa/bQ== 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.1] 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCItgFmMkGAJxAAC4OklsJ9bzbKo0N90xApyQ== 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.2] 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCLtgFmHSP2JRAACe2FmPWFa7JzdonNjKYh6w== 2024-03-25T17:38:41.293 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.3] 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCOtgFm1+sMJBAAzl5auI/6NhouFoC2b9SG5g== 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.x] 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCCtgFmrMKkFRAAbo8jKtFye/tlPv5Kg5YDOw== 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.4] 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCUtgFmXoIkLRAAfJVvghtqiUXpNBWHfuttcQ== 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.5] 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCXtgFmypXlKhAASaHeWJFFqi1DZDQNhYA3IA== 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.6] 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCatgFmjojgKRAAV2pgTQmFwO1zKoM2eBc7sQ== 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.7] 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCdtgFm7U/8JxAAAj4JIOFa+n1/QPnY9w6O7A== 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2024-03-25T17:38:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCCtgFmr/xGGBAAZgI5i5qYHZQfI+i0jZY5eg== 2024-03-25T17:38:41.295 INFO:teuthology.orchestra.run.smithi111.stdout:[client.1] 2024-03-25T17:38:41.295 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCCtgFmguyuGxAAqG7F78yYrdxf9q3UMxnVQg== 2024-03-25T17:38:41.295 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:38:41.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:38:41.390 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.392 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.445 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.447 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.499 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.501 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.552 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.554 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.605 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:38:41.607 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:38:41.659 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.661 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.712 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.715 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.765 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.819 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.821 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:38:41.872 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:38:41.874 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:38:41.928 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:38:41.930 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:38:41.985 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T17:38:41.985 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T17:38:42.011 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:38:42.108 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T17:38:42.134 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T17:38:42.201 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:38:42.297 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T17:38:42.324 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T17:38:42.351 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:38:42.449 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T17:38:42.475 DEBUG:teuthology.orchestra.run.smithi044:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:42.478 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:38:42.531 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T17:38:42.531 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:38:42.531 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T17:38:42.531 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T17:38:42.534 INFO:tasks.ceph.mon.a:Started 2024-03-25T17:38:42.534 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:38:42.534 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T17:38:42.534 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T17:38:42.536 INFO:tasks.ceph.mon.c:Started 2024-03-25T17:38:42.537 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:38:42.537 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T17:38:42.537 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T17:38:42.573 INFO:tasks.ceph.mon.b:Started 2024-03-25T17:38:42.573 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T17:38:42.574 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T17:38:42.574 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T17:38:42.577 INFO:tasks.ceph.mgr.y:Started 2024-03-25T17:38:42.577 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T17:38:42.577 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T17:38:42.579 INFO:tasks.ceph.mgr.x:Started 2024-03-25T17:38:42.579 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T17:38:42.579 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T17:38:42.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.681+0000 7f5d29cae640 10 monclient: get_monmap_and_config 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.681+0000 7f5d29cae640 10 monclient: build_initial_monmap 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.681+0000 7f5d29cae640 10 monclient: monmap: 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 0 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:42.682624+0000 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:42.682624+0000 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:38:42.683 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:38:42.684 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:38:42.684 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:38:42.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.682+0000 7f5d29cae640 1 Processor -- start 2024-03-25T17:38:42.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.682+0000 7f5d29cae640 1 -- start start 2024-03-25T17:38:42.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.682+0000 7f5d29cae640 10 monclient: init 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.682+0000 7f5d29cae640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.682+0000 7f5d29cae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d29cae640 10 monclient(hunting): picked mon.noname-a con 0x7f5d240b4700 addr v1:172.21.15.44:6789/0 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d29cae640 10 monclient(hunting): picked mon.noname-b con 0x7f5d24142a20 addr v1:172.21.15.44:6790/0 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d29cae640 10 monclient(hunting): picked mon.noname-c con 0x7f5d240a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d29cae640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240a9d60 con 0x7f5d240b4700 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d29cae640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2413ff10 con 0x7f5d240a96b0 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d29cae640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240aa1f0 con 0x7f5d24142a20 2024-03-25T17:38:42.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d29cae640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:42.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d29cae640 10 monclient(hunting): authenticate will time out at 5925.910767s 2024-03-25T17:38:42.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d24142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:42.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d22ffd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d2413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:42.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.683+0000 7f5d23fff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d2413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:42.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.884+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d24142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:42.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.884+0000 7f5d22ffd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d2413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:42.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:42.884+0000 7f5d23fff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d2413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:43.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:43.285+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d24142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:43.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:43.285+0000 7f5d22ffd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d2413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:43.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:43.285+0000 7f5d23fff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d2413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:44.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:44.086+0000 7f5d22ffd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d2413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:44.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:44.086+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d24142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:44.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:44.086+0000 7f5d23fff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d2413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:45.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 10 monclient(hunting): tick 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 1 monclient(hunting): continuing hunt 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d2413f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d2413f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d24142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-a con 0x7f5d240b4700 addr v1:172.21.15.44:6789/0 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-b con 0x7f5d24142a20 addr v1:172.21.15.44:6790/0 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-c con 0x7f5d240a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240b4ae0 con 0x7f5d240b4700 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240a9d60 con 0x7f5d240a96b0 2024-03-25T17:38:45.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2413ff10 con 0x7f5d24142a20 2024-03-25T17:38:45.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d217fa640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:45.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d100056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:45.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d22ffd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:45.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.683+0000 7f5d23fff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:45.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.884+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d100056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:45.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.884+0000 7f5d22ffd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:45.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:45.884+0000 7f5d23fff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:46.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:46.285+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d100056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:46.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:46.285+0000 7f5d22ffd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:46.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:46.285+0000 7f5d23fff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:47.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:47.086+0000 7f5d22ffd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:47.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:47.086+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d100056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:47.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:47.086+0000 7f5d23fff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 10 monclient(hunting): tick 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 1 monclient(hunting): continuing hunt 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d100056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-a con 0x7f5d240b4700 addr v1:172.21.15.44:6789/0 2024-03-25T17:38:48.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-c con 0x7f5d240a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:38:48.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-b con 0x7f5d24142a20 addr v1:172.21.15.44:6790/0 2024-03-25T17:38:48.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240aa1f0 con 0x7f5d240b4700 2024-03-25T17:38:48.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240b4ae0 con 0x7f5d240a96b0 2024-03-25T17:38:48.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240a9d60 con 0x7f5d24142a20 2024-03-25T17:38:48.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d217fa640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:48.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d23fff640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:48.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:48.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.683+0000 7f5d22ffd640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:48.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.884+0000 7f5d23fff640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:48.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.884+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:48.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:48.884+0000 7f5d22ffd640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:49.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:49.285+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:49.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:49.285+0000 7f5d23fff640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:49.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:49.285+0000 7f5d22ffd640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:49.548 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:00:06.868 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:49.854 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:00:07.174 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.049 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:00:07.405 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.052 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:00:07.408 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:50.086+0000 7f5d23fff640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T17:38:50.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:50.086+0000 7f5d237fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T17:38:50.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:50.086+0000 7f5d22ffd640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T17:38:50.375 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:00:07.731 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.378 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:00:07.734 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:51.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 10 monclient(hunting): tick 2024-03-25T17:38:51.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:38:51.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 1 monclient(hunting): continuing hunt 2024-03-25T17:38:51.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:38:51.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:51.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d1000c760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:51.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:38:51.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:38:51.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-a con 0x7f5d240b4700 addr v1:172.21.15.44:6789/0 2024-03-25T17:38:51.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-c con 0x7f5d240a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:38:51.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-b con 0x7f5d24142a20 addr v1:172.21.15.44:6790/0 2024-03-25T17:38:51.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2413ff10 con 0x7f5d240b4700 2024-03-25T17:38:51.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240aa1f0 con 0x7f5d240a96b0 2024-03-25T17:38:51.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240b4ae0 con 0x7f5d24142a20 2024-03-25T17:38:51.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.683+0000 7f5d217fa640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:51.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.738+0000 7f5d22ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 0x7f5d10006750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:37704/0 (socket says 172.21.15.44:37704) 2024-03-25T17:38:51.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:51.738+0000 7f5d22ffd640 1 -- 172.21.15.44:0/2683902665 learned_addr learned my addr 172.21.15.44:0/2683902665 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:38:52.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:52.107+0000 7f5d237fe640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T17:38:52.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:52.107+0000 7f5d237fe640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T17:38:52.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:52.107+0000 7f5d237fe640 1 --1- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 0x7f5d1000ffa0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T17:38:52.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:52.139+0000 7f5d23fff640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T17:38:52.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:52.139+0000 7f5d23fff640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T17:38:52.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:52.139+0000 7f5d23fff640 1 --1- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 0x7f5d10008e70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T17:38:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 10 monclient(hunting): tick 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 1 monclient(hunting): continuing hunt 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6790/0 conn(0x7f5d24142a20 legacy=0x7f5d10008e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.111:6789/0 conn(0x7f5d240a96b0 legacy=0x7f5d10006750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-c con 0x7f5d1000c7b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-b con 0x7f5d10012090 addr v1:172.21.15.44:6790/0 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-a con 0x7f5d240b4700 addr v1:172.21.15.44:6789/0 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240a9d60 con 0x7f5d240b4700 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10006750 con 0x7f5d1000c7b0 2024-03-25T17:38:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100078f0 con 0x7f5d10012090 2024-03-25T17:38:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:54.684+0000 7f5d217fa640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:57.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 10 monclient(hunting): tick 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 1 monclient(hunting): continuing hunt 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6790/0 conn(0x7f5d10012090 legacy=0x7f5d10014480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.111:6789/0 conn(0x7f5d1000c7b0 legacy=0x7f5d100107f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6789/0 conn(0x7f5d240b4700 legacy=0x7f5d1000ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-b con 0x7f5d10008c90 addr v1:172.21.15.44:6790/0 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-a con 0x7f5d10008190 addr v1:172.21.15.44:6789/0 2024-03-25T17:38:57.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-c con 0x7f5d1000c7b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:38:57.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10008ae0 con 0x7f5d10008190 2024-03-25T17:38:57.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10009360 con 0x7f5d1000c7b0 2024-03-25T17:38:57.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10008660 con 0x7f5d10008c90 2024-03-25T17:38:57.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:38:57.684+0000 7f5d217fa640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 10 monclient(hunting): tick 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 1 monclient(hunting): continuing hunt 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6790/0 conn(0x7f5d10008c90 legacy=0x7f5d10007a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.111:6789/0 conn(0x7f5d1000c7b0 legacy=0x7f5d100107f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6789/0 conn(0x7f5d10008190 legacy=0x7f5d1000cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-c con 0x7f5d10010020 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-a con 0x7f5d10012470 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:00.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 10 monclient(hunting): picked mon.noname-b con 0x7f5d10008c90 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:00.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10008890 con 0x7f5d10012470 2024-03-25T17:39:00.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10010450 con 0x7f5d10010020 2024-03-25T17:39:00.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100107f0 con 0x7f5d10008c90 2024-03-25T17:39:00.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.684+0000 7f5d217fa640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:00.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.994+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808728738 0 0) 0x7f5d10010450 con 0x7f5d10010020 2024-03-25T17:39:00.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.994+0000 7f5d227fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:00.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.994+0000 7f5d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:00.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.994+0000 7f5d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:00.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.994+0000 7f5d227fc640 10 monclient(hunting): my global_id is 4105 2024-03-25T17:39:00.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:00.994+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d0c003520 con 0x7f5d10010020 2024-03-25T17:39:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.010+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106554951 0 0) 0x7f5d10008890 con 0x7f5d10012470 2024-03-25T17:39:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.010+0000 7f5d227fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.010+0000 7f5d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.010+0000 7f5d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.010+0000 7f5d227fc640 10 monclient(hunting): my global_id is 4104 2024-03-25T17:39:01.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.010+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d10010450 con 0x7f5d10012470 2024-03-25T17:39:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591454462 0 0) 0x7f5d10010450 con 0x7f5d10012470 2024-03-25T17:39:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6790/0 conn(0x7f5d10008c90 legacy=0x7f5d10007a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.111:6789/0 conn(0x7f5d10010020 legacy=0x7f5d1000d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 10 monclient: _finish_hunting 0 2024-03-25T17:39:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 1 monclient: found mon.noname-a 2024-03-25T17:39:01.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10010a20 con 0x7f5d10012470 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 10 monclient: _finish_auth 0 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 10 monclient: _check_auth_tickets 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5d18003e60 con 0x7f5d10012470 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d227fc640 10 monclient: dump: 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:01.026 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:01.027 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d29cae640 5 monclient: authenticate success, global_id 4104 2024-03-25T17:39:01.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.024+0000 7f5d29cae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:01.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.063+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d18004030 con 0x7f5d10012470 2024-03-25T17:39:01.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.063+0000 7f5d227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:01.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d227fc640 1 -- 172.21.15.44:0/2683902665 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5d18004400 con 0x7f5d10012470 2024-03-25T17:39:01.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:01.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 4 set_mon_vals no callback set 2024-03-25T17:39:01.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d227fc640 10 monclient: dump: 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:01.065 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:01.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 10 monclient: shutdown 2024-03-25T17:39:01.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:01.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 1 -- 172.21.15.44:0/2683902665 >> v1:172.21.15.44:6789/0 conn(0x7f5d10012470 legacy=0x7f5d100148c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:01.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 1 -- 172.21.15.44:0/2683902665 shutdown_connections 2024-03-25T17:39:01.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 1 -- 172.21.15.44:0/2683902665 >> 172.21.15.44:0/2683902665 conn(0x7f5d240668e0 msgr2=0x7f5d24066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:01.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 1 -- 172.21.15.44:0/2683902665 shutdown_connections 2024-03-25T17:39:01.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.064+0000 7f5d29cae640 1 -- 172.21.15.44:0/2683902665 wait complete. 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.065+0000 7f5d29cae640 10 monclient: build_initial_monmap 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.065+0000 7f5d29cae640 10 monclient: monmap: 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 0 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:created 0.000000 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.065+0000 7f5d29cae640 1 librados: starting msgr at 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.065+0000 7f5d29cae640 1 librados: starting objecter 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 1 Processor -- start 2024-03-25T17:39:01.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 1 -- start start 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 1 librados: setting wanted keys 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 1 librados: calling monclient init 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 10 monclient: init 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 10 monclient(hunting): picked mon.noname-b con 0x7f5d10008c90 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 10 monclient(hunting): picked mon.noname-c con 0x7f5d10010020 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 10 monclient(hunting): picked mon.noname-a con 0x7f5d10012470 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d241208f0 con 0x7f5d10012470 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d24120b60 con 0x7f5d10008c90 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d241db7d0 con 0x7f5d10010020 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:01.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.066+0000 7f5d29cae640 10 monclient(hunting): authenticate will time out at 5944.294572s 2024-03-25T17:39:01.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.069+0000 7f5d22ffd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f5d10010020 0x7f5d240b0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:52416/0 (socket says 172.21.15.44:52416) 2024-03-25T17:39:01.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.069+0000 7f5d22ffd640 1 -- 172.21.15.44:0/893831421 learned_addr learned my addr 172.21.15.44:0/893831421 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:01.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.069+0000 7f5d0bfff640 10 client.?.objecter ms_handle_connect 0x7f5d10010020 2024-03-25T17:39:01.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.069+0000 7f5d0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:01.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.101+0000 7f5d0bfff640 10 client.?.objecter ms_handle_connect 0x7f5d10012470 2024-03-25T17:39:01.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.101+0000 7f5d0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:01.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.112+0000 7f5d0bfff640 10 client.?.objecter ms_handle_connect 0x7f5d10008c90 2024-03-25T17:39:01.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.112+0000 7f5d0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:01.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.116+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371618198 0 0) 0x7f5d241208f0 con 0x7f5d10012470 2024-03-25T17:39:01.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.116+0000 7f5d0bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:01.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.116+0000 7f5d0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:01.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.116+0000 7f5d0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:01.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.116+0000 7f5d0bfff640 10 monclient(hunting): my global_id is 4110 2024-03-25T17:39:01.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.116+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d10008890 con 0x7f5d10012470 2024-03-25T17:39:01.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.140+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324885573 0 0) 0x7f5d241db7d0 con 0x7f5d10010020 2024-03-25T17:39:01.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.140+0000 7f5d0bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:01.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.140+0000 7f5d0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:01.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.140+0000 7f5d0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:01.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.140+0000 7f5d0bfff640 10 monclient(hunting): my global_id is 4112 2024-03-25T17:39:01.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.140+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d241208f0 con 0x7f5d10010020 2024-03-25T17:39:01.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.156+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3840295592 0 0) 0x7f5d10008890 con 0x7f5d10012470 2024-03-25T17:39:01.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.156+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d241db7d0 con 0x7f5d10012470 2024-03-25T17:39:01.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.156+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5d1c0041a0 con 0x7f5d10012470 2024-03-25T17:39:01.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.156+0000 7f5d0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:01.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.156+0000 7f5d0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.156+0000 7f5d0bfff640 10 monclient(hunting): dump: 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:01.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.156+0000 7f5d0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3060999313 0 0) 0x7f5d241db7d0 con 0x7f5d10012470 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 >> v1:172.21.15.44:6790/0 conn(0x7f5d10010020 legacy=0x7f5d240b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 >> v1:172.21.15.111:6789/0 conn(0x7f5d10008c90 legacy=0x7f5d241daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 1 monclient: found mon.a 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2413a260 con 0x7f5d10012470 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 10 monclient: _finish_auth 0 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 5 monclient: authenticate success, global_id 4110 2024-03-25T17:39:01.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 10 monclient: _renew_subs 2024-03-25T17:39:01.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:01.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2413a510 con 0x7f5d10012470 2024-03-25T17:39:01.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:01.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 10 monclient: _renew_subs 2024-03-25T17:39:01.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:01.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d241e33c0 con 0x7f5d10012470 2024-03-25T17:39:01.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.208+0000 7f5d29cae640 1 librados: init done 2024-03-25T17:39:01.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.209+0000 7f5d29cae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:01.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.209+0000 7f5d29cae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:01.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.209+0000 7f5d29cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:01.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.209+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d241db9c0 con 0x7f5d10012470 2024-03-25T17:39:01.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.213+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d1c0045b0 con 0x7f5d10012470 2024-03-25T17:39:01.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.213+0000 7f5d0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:01.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.213+0000 7f5d21ffb640 4 set_mon_vals no callback set 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.217+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5d1c004a00 con 0x7f5d10012470 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.217+0000 7f5d0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.217+0000 7f5d0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.217+0000 7f5d0bfff640 10 monclient: dump: 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:01.219 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:01.219 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5d1c004f70 con 0x7f5d10012470 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2693561893 0 0) 0x7f5d1c006250 con 0x7f5d10012470 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 10 client.4110.objecter ms_dispatch 0x7f5d24000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5d1c006970 con 0x7f5d10012470 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:01.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.425+0000 7f5d0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:01.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.493+0000 7f5d29cae640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:39:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.493+0000 7f5d29cae640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:39:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.493+0000 7f5d29cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:01.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.493+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5d241db9c0 con 0x7f5d10012470 2024-03-25T17:39:01.654 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:01.653+0000 7f9ccb458040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T17:39:01.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.675+0000 7f5d0bfff640 1 -- 172.21.15.44:0/893831421 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f5d1c005530 con 0x7f5d10012470 2024-03-25T17:39:01.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.675+0000 7f5d0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:39:01.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.675+0000 7f5d0bfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T17:39:01.676 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2024-03-25T17:39:01.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.676+0000 7f5d29cae640 10 librados: watch_flush enter 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.676+0000 7f5d29cae640 10 librados: watch_flush exit 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.676+0000 7f5d29cae640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.676+0000 7f5d29cae640 10 client.4110.objecter successfully canceled tick 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.677+0000 7f5d29cae640 10 monclient: shutdown 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.677+0000 7f5d29cae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.677+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 >> v1:172.21.15.44:6789/0 conn(0x7f5d10012470 legacy=0x7f5d240b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.677+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 shutdown_connections 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.677+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 >> 172.21.15.44:0/893831421 conn(0x7f5d240668e0 msgr2=0x7f5d240b2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.677+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 shutdown_connections 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.677+0000 7f5d29cae640 1 -- 172.21.15.44:0/893831421 wait complete. 2024-03-25T17:39:01.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:01.677+0000 7f5d29cae640 1 librados: shutdown 2024-03-25T17:39:01.688 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T17:39:01.688 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T17:39:01.689 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:39:01.689 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T17:39:01.714 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:01.714+0000 7f9ccb458040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T17:39:01.718 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:39:01.718 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T17:39:01.777 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:01.777+0000 7fca18a2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T17:39:01.786 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:39:01.786 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T17:39:01.831 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:01.831+0000 7fca18a2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T17:39:01.854 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T17:39:01.854 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T17:39:01.921 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:39:01.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T17:39:01.948 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:39:01.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T17:39:02.016 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:39:02.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T17:39:02.083 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T17:39:02.083 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T17:39:02.149 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 620cf0f3-0d96-4974-984e-50066cd7e73d 0 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.290+0000 7f4b39d4d640 10 monclient: get_monmap_and_config 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.290+0000 7f4b39d4d640 10 monclient: build_initial_monmap 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.290+0000 7f4b39d4d640 10 monclient: monmap: 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:39:02.291893+0000 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:39:02.291893+0000 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.292 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:02.292 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:02.292 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:02.292 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.293 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.291+0000 7f4b39d4d640 1 Processor -- start 2024-03-25T17:39:02.293 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.291+0000 7f4b39d4d640 1 -- start start 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.291+0000 7f4b39d4d640 10 monclient: init 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f4b340b4700 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f4b34142a20 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f4b340a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b340a9d60 con 0x7f4b34142a20 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3413ff10 con 0x7f4b340a96b0 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b340aa1f0 con 0x7f4b340b4700 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b39d4d640 10 monclient(hunting): authenticate will time out at 5937.428163s 2024-03-25T17:39:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.292+0000 7f4b32ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4b34142a20 0x7f4b3413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33778/0 (socket says 172.21.15.111:33778) 2024-03-25T17:39:02.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.293+0000 7f4b32ffd640 1 -- 172.21.15.111:0/2648202916 learned_addr learned my addr 172.21.15.111:0/2648202916 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:02.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.295+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699746032 0 0) 0x7f4b340a9d60 con 0x7f4b34142a20 2024-03-25T17:39:02.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.295+0000 7f4b327fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.295+0000 7f4b327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.295+0000 7f4b327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.295+0000 7f4b327fc640 10 monclient(hunting): my global_id is 4116 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.295+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b20003520 con 0x7f4b34142a20 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150727055 0 0) 0x7f4b3413ff10 con 0x7f4b340a96b0 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 10 monclient(hunting): my global_id is 4114 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b340a9d60 con 0x7f4b340a96b0 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900378047 0 0) 0x7f4b340aa1f0 con 0x7f4b340b4700 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 10 monclient(hunting): my global_id is 4115 2024-03-25T17:39:02.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.296+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3413ff10 con 0x7f4b340b4700 2024-03-25T17:39:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351268183 0 0) 0x7f4b20003520 con 0x7f4b34142a20 2024-03-25T17:39:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 >> v1:172.21.15.44:6790/0 conn(0x7f4b340b4700 legacy=0x7f4b34142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 >> v1:172.21.15.111:6789/0 conn(0x7f4b340a96b0 legacy=0x7f4b3413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 10 monclient: _finish_hunting 0 2024-03-25T17:39:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 1 monclient: found mon.noname-a 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b340b4ae0 con 0x7f4b34142a20 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 10 monclient: _finish_auth 0 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 10 monclient: _check_auth_tickets 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4b1c001690 con 0x7f4b34142a20 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b327fc640 10 monclient: dump: 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b39d4d640 5 monclient: authenticate success, global_id 4116 2024-03-25T17:39:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.299+0000 7f4b39d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.301+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b1c0017f0 con 0x7f4b34142a20 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.301+0000 7f4b327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.301+0000 7f4b327fc640 1 -- 172.21.15.111:0/2648202916 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4b1c001c00 con 0x7f4b34142a20 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.301+0000 7f4b39d4d640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.301+0000 7f4b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.301+0000 7f4b39d4d640 4 set_mon_vals no callback set 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.301+0000 7f4b327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.301+0000 7f4b327fc640 10 monclient: dump: 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.302+0000 7f4b39d4d640 10 monclient: shutdown 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.302+0000 7f4b39d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.302+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2648202916 >> v1:172.21.15.44:6789/0 conn(0x7f4b34142a20 legacy=0x7f4b3413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.302+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2648202916 shutdown_connections 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.302+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2648202916 >> 172.21.15.111:0/2648202916 conn(0x7f4b340668e0 msgr2=0x7f4b34066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.302+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2648202916 shutdown_connections 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.302+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2648202916 wait complete. 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 10 monclient: build_initial_monmap 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 10 monclient: monmap: 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 1 librados: starting msgr at 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 1 librados: starting objecter 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 1 Processor -- start 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 1 -- start start 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 1 librados: setting wanted keys 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 1 librados: calling monclient init 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.303+0000 7f4b39d4d640 10 monclient: init 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f4b340a96b0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f4b340b4700 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f4b34142a20 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3413c170 con 0x7f4b340b4700 2024-03-25T17:39:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b340abf30 con 0x7f4b34142a20 2024-03-25T17:39:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3413c600 con 0x7f4b340a96b0 2024-03-25T17:39:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b39d4d640 10 monclient(hunting): authenticate will time out at 5937.440266s 2024-03-25T17:39:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b32ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f4b340b4700 0x7f4b34131b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33784/0 (socket says 172.21.15.111:33784) 2024-03-25T17:39:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.304+0000 7f4b32ffd640 1 -- 172.21.15.111:0/2251367276 learned_addr learned my addr 172.21.15.111:0/2251367276 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.305+0000 7f4b2bfff640 10 client.?.objecter ms_handle_connect 0x7f4b340b4700 2024-03-25T17:39:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.305+0000 7f4b2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:02.311 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.308+0000 7f9ccb458040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T17:39:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.307+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706989183 0 0) 0x7f4b3413c170 con 0x7f4b340b4700 2024-03-25T17:39:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.307+0000 7f4b2bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.307+0000 7f4b2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.307+0000 7f4b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.307+0000 7f4b2bfff640 10 monclient(hunting): my global_id is 4119 2024-03-25T17:39:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.307+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b14004eb0 con 0x7f4b340b4700 2024-03-25T17:39:02.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.317+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282225506 0 0) 0x7f4b14004eb0 con 0x7f4b340b4700 2024-03-25T17:39:02.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.317+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b3413c170 con 0x7f4b340b4700 2024-03-25T17:39:02.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.317+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4b1c001bc0 con 0x7f4b340b4700 2024-03-25T17:39:02.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.317+0000 7f4b2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.317+0000 7f4b2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.317+0000 7f4b2bfff640 10 monclient(hunting): dump: 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.317+0000 7f4b2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2560839201 0 0) 0x7f4b3413c170 con 0x7f4b340b4700 2024-03-25T17:39:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 >> v1:172.21.15.44:6790/0 conn(0x7f4b340a96b0 legacy=0x7f4b341387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 >> v1:172.21.15.111:6789/0 conn(0x7f4b34142a20 legacy=0x7f4b34132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:39:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 1 monclient: found mon.a 2024-03-25T17:39:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34132980 con 0x7f4b340b4700 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 10 monclient: _finish_auth 0 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.321+0000 7f4b2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 5 monclient: authenticate success, global_id 4119 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 10 monclient: _renew_subs 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3413cb70 con 0x7f4b340b4700 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 10 monclient: _renew_subs 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b341390b0 con 0x7f4b340b4700 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 1 librados: init done 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:02.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.322+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3404f730 con 0x7f4b340b4700 2024-03-25T17:39:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b1c004a90 con 0x7f4b340b4700 2024-03-25T17:39:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4b1c004ea0 con 0x7f4b340b4700 2024-03-25T17:39:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 10 monclient: dump: 2024-03-25T17:39:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4b1c005450 con 0x7f4b340b4700 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b31ffb640 4 set_mon_vals no callback set 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3240638899 0 0) 0x7f4b1c0065b0 con 0x7f4b340b4700 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 10 client.4119.objecter ms_dispatch 0x7f4b34000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T17:39:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T17:39:02.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T17:39:02.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.327+0000 7f4b2bfff640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T17:39:02.368 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.368+0000 7f9ccb458040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T17:39:02.384 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.384+0000 7fca18a2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T17:39:02.414 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.414+0000 7f9ccb458040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T17:39:02.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.435+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4b1c005700 con 0x7f4b340b4700 2024-03-25T17:39:02.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.435+0000 7f4b2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:02.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.435+0000 7f4b2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:02.441 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.440+0000 7fca18a2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T17:39:02.471 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.470+0000 7f9ccb458040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T17:39:02.487 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.487+0000 7fca18a2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T17:39:02.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.508+0000 7f4b39d4d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "620cf0f3-0d96-4974-984e-50066cd7e73d", "id": 0}] 2024-03-25T17:39:02.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.508+0000 7f4b39d4d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "620cf0f3-0d96-4974-984e-50066cd7e73d", "id": 0}] 2024-03-25T17:39:02.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.508+0000 7f4b39d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.509+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd new", "uuid": "620cf0f3-0d96-4974-984e-50066cd7e73d", "id": 0} v 0) v1 -- 0x7f4b340b77c0 con 0x7f4b340b4700 2024-03-25T17:39:02.514 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.514+0000 7f9ccb458040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T17:39:02.544 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.544+0000 7fca18a2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T17:39:02.589 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.588+0000 7fca18a2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T17:39:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.600+0000 7f4b2bfff640 1 -- 172.21.15.111:0/2251367276 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "620cf0f3-0d96-4974-984e-50066cd7e73d", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2548854663 0 732707515) 0x7f4b1c005990 con 0x7f4b340b4700 2024-03-25T17:39:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.601+0000 7f4b2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "620cf0f3-0d96-4974-984e-50066cd7e73d", "id": 0}] 2024-03-25T17:39:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.601+0000 7f4b2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:02.602 INFO:teuthology.orchestra.run.smithi111.stdout:0 2024-03-25T17:39:02.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.601+0000 7f4b39d4d640 10 librados: watch_flush enter 2024-03-25T17:39:02.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.601+0000 7f4b39d4d640 10 librados: watch_flush exit 2024-03-25T17:39:02.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.601+0000 7f4b39d4d640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T17:39:02.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.601+0000 7f4b39d4d640 10 client.4119.objecter successfully canceled tick 2024-03-25T17:39:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.602+0000 7f4b39d4d640 10 monclient: shutdown 2024-03-25T17:39:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.602+0000 7f4b39d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.602+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 >> v1:172.21.15.44:6789/0 conn(0x7f4b340b4700 legacy=0x7f4b34131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.602+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 shutdown_connections 2024-03-25T17:39:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.602+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 >> 172.21.15.111:0/2251367276 conn(0x7f4b340668e0 msgr2=0x7f4b340b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.602+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 shutdown_connections 2024-03-25T17:39:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.602+0000 7f4b39d4d640 1 -- 172.21.15.111:0/2251367276 wait complete. 2024-03-25T17:39:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.602+0000 7f4b39d4d640 1 librados: shutdown 2024-03-25T17:39:02.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new e8b5557d-5398-45c4-8d27-c70dce8bdbea 1 2024-03-25T17:39:02.638 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.638+0000 7f9ccb458040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T17:39:02.711 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.710+0000 7f9ccb458040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T17:39:02.715 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.715+0000 7fca18a2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T17:39:02.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.723+0000 7febc1bed640 10 monclient: get_monmap_and_config 2024-03-25T17:39:02.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.723+0000 7febc1bed640 10 monclient: build_initial_monmap 2024-03-25T17:39:02.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.723+0000 7febc1bed640 10 monclient: monmap: 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:39:02.724540+0000 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:39:02.724540+0000 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:02.724 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.724+0000 7febc1bed640 1 Processor -- start 2024-03-25T17:39:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.724+0000 7febc1bed640 1 -- start start 2024-03-25T17:39:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.724+0000 7febc1bed640 10 monclient: init 2024-03-25T17:39:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.724+0000 7febc1bed640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.724+0000 7febc1bed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.724+0000 7febc1bed640 10 monclient(hunting): picked mon.noname-b con 0x7febbc0a9920 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febc1bed640 10 monclient(hunting): picked mon.noname-a con 0x7febbc0b45a0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febc1bed640 10 monclient(hunting): picked mon.noname-c con 0x7febbc146450 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febc1bed640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc06b100 con 0x7febbc0b45a0 2024-03-25T17:39:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febc1bed640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc0b2940 con 0x7febbc146450 2024-03-25T17:39:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febc1bed640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc13fec0 con 0x7febbc0a9920 2024-03-25T17:39:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febc1bed640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febc1bed640 10 monclient(hunting): authenticate will time out at 5937.860786s 2024-03-25T17:39:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febbaffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7febbc0b45a0 0x7febbc145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33800/0 (socket says 172.21.15.111:33800) 2024-03-25T17:39:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.725+0000 7febbaffd640 1 -- 172.21.15.111:0/723536431 learned_addr learned my addr 172.21.15.111:0/723536431 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:02.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283728155 0 0) 0x7febbc06b100 con 0x7febbc0b45a0 2024-03-25T17:39:02.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): my global_id is 4122 2024-03-25T17:39:02.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba8003520 con 0x7febbc0b45a0 2024-03-25T17:39:02.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156246405 0 0) 0x7febbc0b2940 con 0x7febbc146450 2024-03-25T17:39:02.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): my global_id is 4117 2024-03-25T17:39:02.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc06b100 con 0x7febbc146450 2024-03-25T17:39:02.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671920251 0 0) 0x7febbc13fec0 con 0x7febbc0a9920 2024-03-25T17:39:02.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 10 monclient(hunting): my global_id is 4118 2024-03-25T17:39:02.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.728+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc0b2940 con 0x7febbc0a9920 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2668196825 0 0) 0x7feba8003520 con 0x7febbc0b45a0 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 >> v1:172.21.15.44:6790/0 conn(0x7febbc0a9920 legacy=0x7febbc13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 >> v1:172.21.15.111:6789/0 conn(0x7febbc146450 legacy=0x7febbc14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 1 monclient: found mon.noname-a 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc140050 con 0x7febbc0b45a0 2024-03-25T17:39:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 10 monclient: _finish_auth 0 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7feba4001690 con 0x7febbc0b45a0 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febba7fc640 10 monclient: dump: 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.734 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.734 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:02.734 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:02.734 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:02.734 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febc1bed640 5 monclient: authenticate success, global_id 4122 2024-03-25T17:39:02.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.731+0000 7febc1bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:02.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.732+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba40017f0 con 0x7febbc0b45a0 2024-03-25T17:39:02.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.732+0000 7febba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:02.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.732+0000 7febc1bed640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:02.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.732+0000 7febc1bed640 4 set_mon_vals no callback set 2024-03-25T17:39:02.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febba7fc640 1 -- 172.21.15.111:0/723536431 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7feba4001a60 con 0x7febbc0b45a0 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febba7fc640 10 monclient: dump: 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:02.738 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febc1bed640 10 monclient: shutdown 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febc1bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febc1bed640 1 -- 172.21.15.111:0/723536431 >> v1:172.21.15.44:6789/0 conn(0x7febbc0b45a0 legacy=0x7febbc145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febc1bed640 1 -- 172.21.15.111:0/723536431 shutdown_connections 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febc1bed640 1 -- 172.21.15.111:0/723536431 >> 172.21.15.111:0/723536431 conn(0x7febbc0b7f40 msgr2=0x7febbc0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.733+0000 7febc1bed640 1 -- 172.21.15.111:0/723536431 shutdown_connections 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.734+0000 7febc1bed640 1 -- 172.21.15.111:0/723536431 wait complete. 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.734+0000 7febc1bed640 10 monclient: build_initial_monmap 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.734+0000 7febc1bed640 10 monclient: monmap: 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T17:39:02.739 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.735+0000 7febc1bed640 1 librados: starting msgr at 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.735+0000 7febc1bed640 1 librados: starting objecter 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.735+0000 7febc1bed640 1 Processor -- start 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.735+0000 7febc1bed640 1 -- start start 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.735+0000 7febc1bed640 1 librados: setting wanted keys 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.735+0000 7febc1bed640 1 librados: calling monclient init 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.735+0000 7febc1bed640 10 monclient: init 2024-03-25T17:39:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 10 monclient(hunting): picked mon.noname-a con 0x7febbc0a9920 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 10 monclient(hunting): picked mon.noname-b con 0x7febbc0b45a0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 10 monclient(hunting): picked mon.noname-c con 0x7febbc146450 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc1d8400 con 0x7febbc0a9920 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc0ac400 con 0x7febbc0b45a0 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc0ac650 con 0x7febbc146450 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.736+0000 7febc1bed640 10 monclient(hunting): authenticate will time out at 5937.872443s 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.737+0000 7febbb7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7febbc0a9920 0x7febbc1d7a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33814/0 (socket says 172.21.15.111:33814) 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.737+0000 7febbb7fe640 1 -- 172.21.15.111:0/3862858069 learned_addr learned my addr 172.21.15.111:0/3862858069 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.737+0000 7febb3fff640 10 client.?.objecter ms_handle_connect 0x7febbc0a9920 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.737+0000 7febb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.737+0000 7febb3fff640 10 client.?.objecter ms_handle_connect 0x7febbc146450 2024-03-25T17:39:02.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.737+0000 7febb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.738+0000 7febb3fff640 10 client.?.objecter ms_handle_connect 0x7febbc0b45a0 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.738+0000 7febb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277383374 0 0) 0x7febbc1d8400 con 0x7febbc0a9920 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 10 monclient(hunting): my global_id is 4125 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c004eb0 con 0x7febbc0a9920 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233982345 0 0) 0x7febbc0ac650 con 0x7febbc146450 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 10 monclient(hunting): my global_id is 4121 2024-03-25T17:39:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.739+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc1d8400 con 0x7febbc146450 2024-03-25T17:39:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.740+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983505504 0 0) 0x7febbc0ac400 con 0x7febbc0b45a0 2024-03-25T17:39:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.740+0000 7febb3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.740+0000 7febb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.740+0000 7febb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.740+0000 7febb3fff640 10 monclient(hunting): my global_id is 4120 2024-03-25T17:39:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.740+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc0ac650 con 0x7febbc0b45a0 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.744+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1123971658 0 0) 0x7feb9c004eb0 con 0x7febbc0a9920 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.744+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febbc0ac400 con 0x7febbc0a9920 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.744+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7febac001d90 con 0x7febbc0a9920 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.744+0000 7febb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.744+0000 7febb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.744+0000 7febb3fff640 10 monclient(hunting): dump: 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.746 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.746 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.746 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.746 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:02.746 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:02.746 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:02.746 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.744+0000 7febb3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:02.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1298662077 0 0) 0x7febbc0ac400 con 0x7febbc0a9920 2024-03-25T17:39:02.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 >> v1:172.21.15.44:6790/0 conn(0x7febbc146450 legacy=0x7febbc140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 >> v1:172.21.15.111:6789/0 conn(0x7febbc0b45a0 legacy=0x7febbc0b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 1 monclient: found mon.a 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc0b18f0 con 0x7febbc0a9920 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 10 monclient: _finish_auth 0 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 5 monclient: authenticate success, global_id 4125 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 10 monclient: _renew_subs 2024-03-25T17:39:02.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febbc0b1ba0 con 0x7febbc0a9920 2024-03-25T17:39:02.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:02.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 10 monclient: _renew_subs 2024-03-25T17:39:02.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febbc13c3c0 con 0x7febbc0a9920 2024-03-25T17:39:02.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.748+0000 7febc1bed640 1 librados: init done 2024-03-25T17:39:02.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.749+0000 7febc1bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:02.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.749+0000 7febc1bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:02.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.749+0000 7febc1bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.749+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febbc1d8160 con 0x7febbc0a9920 2024-03-25T17:39:02.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febac001ff0 con 0x7febbc0a9920 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7febac003de0 con 0x7febbc0a9920 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 10 monclient: dump: 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb9ffb640 4 set_mon_vals no callback set 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7febac004390 con 0x7febbc0a9920 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 329559209 0 0) 0x7febac006a40 con 0x7febbc0a9920 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 10 client.4125.objecter ms_dispatch 0x7febbc000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T17:39:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.752+0000 7febb3fff640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T17:39:02.763 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.763+0000 7f9ccb458040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T17:39:02.790 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.790+0000 7fca18a2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T17:39:02.817 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.817+0000 7f9ccb458040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T17:39:02.837 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.837+0000 7fca18a2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T17:39:02.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.855+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7febac004640 con 0x7febbc0a9920 2024-03-25T17:39:02.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.856+0000 7febb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:02.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.856+0000 7febb3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:02.869 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:02.868+0000 7f9ccb458040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T17:39:02.892 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.892+0000 7fca18a2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T17:39:02.941 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:02.941+0000 7fca18a2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T17:39:02.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.941+0000 7febc1bed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e8b5557d-5398-45c4-8d27-c70dce8bdbea", "id": 1}] 2024-03-25T17:39:02.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.941+0000 7febc1bed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e8b5557d-5398-45c4-8d27-c70dce8bdbea", "id": 1}] 2024-03-25T17:39:02.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.941+0000 7febc1bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:02.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.941+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e8b5557d-5398-45c4-8d27-c70dce8bdbea", "id": 1} v 0) v1 -- 0x7febbc04f730 con 0x7febbc0a9920 2024-03-25T17:39:02.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.997+0000 7febb3fff640 1 -- 172.21.15.111:0/3862858069 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8b5557d-5398-45c4-8d27-c70dce8bdbea", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2112821556 0 940483276) 0x7febac005b40 con 0x7febbc0a9920 2024-03-25T17:39:02.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.997+0000 7febb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e8b5557d-5398-45c4-8d27-c70dce8bdbea", "id": 1}] 2024-03-25T17:39:02.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.997+0000 7febb3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:02.997 INFO:teuthology.orchestra.run.smithi111.stdout:1 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 10 librados: watch_flush enter 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 10 librados: watch_flush exit 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 10 client.4125.objecter successfully canceled tick 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 10 monclient: shutdown 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 >> v1:172.21.15.44:6789/0 conn(0x7febbc0a9920 legacy=0x7febbc1d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 shutdown_connections 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 >> 172.21.15.111:0/3862858069 conn(0x7febbc0b7f40 msgr2=0x7febbc138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 shutdown_connections 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 1 -- 172.21.15.111:0/3862858069 wait complete. 2024-03-25T17:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:02.998+0000 7febc1bed640 1 librados: shutdown 2024-03-25T17:39:03.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5 2 2024-03-25T17:39:03.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.130+0000 7f431e93d640 10 monclient: get_monmap_and_config 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.130+0000 7f431e93d640 10 monclient: build_initial_monmap 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.130+0000 7f431e93d640 10 monclient: monmap: 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:39:03.131633+0000 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:39:03.131633+0000 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:03.131 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.131+0000 7f431e93d640 1 Processor -- start 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.131+0000 7f431e93d640 1 -- start start 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.131+0000 7f431e93d640 10 monclient: init 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 10 monclient(hunting): picked mon.noname-c con 0x7f43180a9920 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 10 monclient(hunting): picked mon.noname-a con 0x7f43180b45a0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 10 monclient(hunting): picked mon.noname-b con 0x7f4318146450 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431806b100 con 0x7f43180b45a0 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180b2940 con 0x7f43180a9920 2024-03-25T17:39:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431813fec0 con 0x7f4318146450 2024-03-25T17:39:03.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:03.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f431e93d640 10 monclient(hunting): authenticate will time out at 5938.267902s 2024-03-25T17:39:03.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f43177fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f43180b45a0 0x7f4318145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33824/0 (socket says 172.21.15.111:33824) 2024-03-25T17:39:03.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.132+0000 7f43177fe640 1 -- 172.21.15.111:0/1005557022 learned_addr learned my addr 172.21.15.111:0/1005557022 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218722199 0 0) 0x7f431806b100 con 0x7f43180b45a0 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): my global_id is 4128 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4300003520 con 0x7f43180b45a0 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478706091 0 0) 0x7f43180b2940 con 0x7f43180a9920 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): my global_id is 4123 2024-03-25T17:39:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f431806b100 con 0x7f43180a9920 2024-03-25T17:39:03.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528037608 0 0) 0x7f431813fec0 con 0x7f4318146450 2024-03-25T17:39:03.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 10 monclient(hunting): my global_id is 4124 2024-03-25T17:39:03.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.135+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43180b2940 con 0x7f4318146450 2024-03-25T17:39:03.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 485102759 0 0) 0x7f431806b100 con 0x7f43180a9920 2024-03-25T17:39:03.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 >> v1:172.21.15.44:6790/0 conn(0x7f4318146450 legacy=0x7f431814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 >> v1:172.21.15.44:6789/0 conn(0x7f43180b45a0 legacy=0x7f4318145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 1 monclient: found mon.noname-c 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318140050 con 0x7f43180a9920 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4308001670 con 0x7f43180a9920 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.138+0000 7f4316ffd640 10 monclient: dump: 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.139+0000 7f431e93d640 5 monclient: authenticate success, global_id 4123 2024-03-25T17:39:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.139+0000 7f431e93d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.166+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43080017d0 con 0x7f43180a9920 2024-03-25T17:39:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.166+0000 7f4316ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.166+0000 7f4316ffd640 1 -- 172.21.15.111:0/1005557022 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4308001a40 con 0x7f43180a9920 2024-03-25T17:39:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.166+0000 7f4316ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.166+0000 7f4316ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.166+0000 7f4316ffd640 10 monclient: dump: 2024-03-25T17:39:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.166+0000 7f431e93d640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.166+0000 7f431e93d640 4 set_mon_vals no callback set 2024-03-25T17:39:03.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.167+0000 7f431e93d640 10 monclient: shutdown 2024-03-25T17:39:03.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.167+0000 7f431e93d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:03.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.167+0000 7f431e93d640 1 -- 172.21.15.111:0/1005557022 >> v1:172.21.15.111:6789/0 conn(0x7f43180a9920 legacy=0x7f431813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.167+0000 7f431e93d640 1 -- 172.21.15.111:0/1005557022 shutdown_connections 2024-03-25T17:39:03.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.167+0000 7f431e93d640 1 -- 172.21.15.111:0/1005557022 >> 172.21.15.111:0/1005557022 conn(0x7f43180b7f40 msgr2=0x7f43180b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:03.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.167+0000 7f431e93d640 1 -- 172.21.15.111:0/1005557022 shutdown_connections 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.167+0000 7f431e93d640 1 -- 172.21.15.111:0/1005557022 wait complete. 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 10 monclient: build_initial_monmap 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 10 monclient: monmap: 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T17:39:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 1 librados: starting msgr at 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 1 librados: starting objecter 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 1 Processor -- start 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 1 -- start start 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 1 librados: setting wanted keys 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 1 librados: calling monclient init 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.168+0000 7f431e93d640 10 monclient: init 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 10 monclient(hunting): picked mon.noname-a con 0x7f43180a9920 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 10 monclient(hunting): picked mon.noname-c con 0x7f43180b45a0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 10 monclient(hunting): picked mon.noname-b con 0x7f4318146450 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180ac0e0 con 0x7f43180a9920 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180ac330 con 0x7f4318146450 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4318143fe0 con 0x7f43180b45a0 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.169+0000 7f431e93d640 10 monclient(hunting): authenticate will time out at 5938.305291s 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.170+0000 7f4317fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f43180a9920 0x7f4318144500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33832/0 (socket says 172.21.15.111:33832) 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.170+0000 7f4317fff640 1 -- 172.21.15.111:0/605868649 learned_addr learned my addr 172.21.15.111:0/605868649 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.170+0000 7f4314ff9640 10 client.?.objecter ms_handle_connect 0x7f43180a9920 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.170+0000 7f4314ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.170+0000 7f4314ff9640 10 client.?.objecter ms_handle_connect 0x7f43180b45a0 2024-03-25T17:39:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.170+0000 7f4314ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:03.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.171+0000 7f4314ff9640 10 client.?.objecter ms_handle_connect 0x7f4318146450 2024-03-25T17:39:03.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.171+0000 7f4314ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.172+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691849500 0 0) 0x7f43180ac0e0 con 0x7f43180a9920 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.172+0000 7f4314ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.172+0000 7f4314ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.172+0000 7f4314ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.172+0000 7f4314ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.172+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4304004eb0 con 0x7f43180a9920 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208559532 0 0) 0x7f4318143fe0 con 0x7f43180b45a0 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 10 monclient(hunting): my global_id is 4127 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43180ac0e0 con 0x7f43180b45a0 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898584018 0 0) 0x7f43180ac330 con 0x7f4318146450 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 10 monclient(hunting): my global_id is 4126 2024-03-25T17:39:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.173+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4318143fe0 con 0x7f4318146450 2024-03-25T17:39:03.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.178+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3146253154 0 0) 0x7f4304004eb0 con 0x7f43180a9920 2024-03-25T17:39:03.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.179+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43180ac330 con 0x7f43180a9920 2024-03-25T17:39:03.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.179+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4308001ae0 con 0x7f43180a9920 2024-03-25T17:39:03.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.179+0000 7f4314ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.179+0000 7f4314ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:03.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.179+0000 7f4314ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.179+0000 7f4314ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:03.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.179+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1563781248 0 0) 0x7f43180ac0e0 con 0x7f43180b45a0 2024-03-25T17:39:03.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.180+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4304004eb0 con 0x7f43180b45a0 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.180+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f430c001950 con 0x7f43180b45a0 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.180+0000 7f4314ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.180+0000 7f4314ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.180+0000 7f4314ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.181+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1165747393 0 0) 0x7f4318143fe0 con 0x7f4318146450 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.181+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43180ac0e0 con 0x7f4318146450 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.181+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4310001bd0 con 0x7f4318146450 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.181+0000 7f4314ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.181+0000 7f4314ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.181+0000 7f4314ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.183 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.184 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.184 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.184 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3330847720 0 0) 0x7f43180ac330 con 0x7f43180a9920 2024-03-25T17:39:03.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 >> v1:172.21.15.44:6790/0 conn(0x7f43180b45a0 legacy=0x7f4318137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 >> v1:172.21.15.111:6789/0 conn(0x7f4318146450 legacy=0x7f43181e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 1 monclient: found mon.a 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318144c10 con 0x7f43180a9920 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 10 monclient: _finish_auth 0 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f4314ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 5 monclient: authenticate success, global_id 4131 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 10 monclient: _renew_subs 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4318141340 con 0x7f43180a9920 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 10 monclient: _renew_subs 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43181db990 con 0x7f43180a9920 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.183+0000 7f431e93d640 1 librados: init done 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.184+0000 7f431e93d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.184+0000 7f431e93d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.184+0000 7f431e93d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.184+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431804f730 con 0x7f43180a9920 2024-03-25T17:39:03.276 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:03.276+0000 7f9ccb458040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.289+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43080038a0 con 0x7f43180a9920 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.289+0000 7f4314ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.289+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f4308003cb0 con 0x7f43180a9920 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.289+0000 7f4314ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.289+0000 7f4314ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.289+0000 7f4314ff9640 10 monclient: dump: 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.290 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.289+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4308004260 con 0x7f43180a9920 2024-03-25T17:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.289+0000 7f43167fc640 4 set_mon_vals no callback set 2024-03-25T17:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.290+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3001348816 0 0) 0x7f43080067f0 con 0x7f43180a9920 2024-03-25T17:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.290+0000 7f4314ff9640 10 client.4131.objecter ms_dispatch 0x7f4318000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T17:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.290+0000 7f4314ff9640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T17:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.290+0000 7f4314ff9640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T17:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.290+0000 7f4314ff9640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T17:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.290+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f43080063b0 con 0x7f43180a9920 2024-03-25T17:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.290+0000 7f4314ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.290+0000 7f4314ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:03.349 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:03.349+0000 7fca18a2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T17:39:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.362+0000 7f431e93d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5", "id": 2}] 2024-03-25T17:39:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.362+0000 7f431e93d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5", "id": 2}] 2024-03-25T17:39:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.363+0000 7f431e93d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.363+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5", "id": 2} v 0) v1 -- 0x7f43180b62c0 con 0x7f43180a9920 2024-03-25T17:39:03.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.419+0000 7f4314ff9640 1 -- 172.21.15.111:0/605868649 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3174670841 0 216599125) 0x7f4308006ae0 con 0x7f43180a9920 2024-03-25T17:39:03.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.419+0000 7f4314ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5", "id": 2}] 2024-03-25T17:39:03.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.419+0000 7f4314ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:03.420 INFO:teuthology.orchestra.run.smithi111.stdout:2 2024-03-25T17:39:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 10 librados: watch_flush enter 2024-03-25T17:39:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 10 librados: watch_flush exit 2024-03-25T17:39:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T17:39:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 10 client.4131.objecter successfully canceled tick 2024-03-25T17:39:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 10 monclient: shutdown 2024-03-25T17:39:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 >> v1:172.21.15.44:6789/0 conn(0x7f43180a9920 legacy=0x7f4318144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 shutdown_connections 2024-03-25T17:39:03.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 >> 172.21.15.111:0/605868649 conn(0x7f43180b7f40 msgr2=0x7f4318074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:03.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 shutdown_connections 2024-03-25T17:39:03.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 1 -- 172.21.15.111:0/605868649 wait complete. 2024-03-25T17:39:03.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.420+0000 7f431e93d640 1 librados: shutdown 2024-03-25T17:39:03.422 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:03.419+0000 7f9ccb458040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T17:39:03.431 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new ed0322fe-4a1a-4b8c-9886-9d7b9a18530e 3 2024-03-25T17:39:03.492 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:03.492+0000 7fca18a2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T17:39:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.530+0000 7f0a0ed7b640 10 monclient: get_monmap_and_config 2024-03-25T17:39:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.530+0000 7f0a0ed7b640 10 monclient: build_initial_monmap 2024-03-25T17:39:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.530+0000 7f0a0ed7b640 10 monclient: monmap: 2024-03-25T17:39:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:39:03.531814+0000 2024-03-25T17:39:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:39:03.531814+0000 2024-03-25T17:39:03.532 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:03.532 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.532 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:03.532 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:03.532 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:03.532 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.531+0000 7f0a0ed7b640 1 Processor -- start 2024-03-25T17:39:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.531+0000 7f0a0ed7b640 1 -- start start 2024-03-25T17:39:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.531+0000 7f0a0ed7b640 10 monclient: init 2024-03-25T17:39:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 10 monclient(hunting): picked mon.noname-b con 0x7f0a080b4700 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 10 monclient(hunting): picked mon.noname-c con 0x7f0a08142a20 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 10 monclient(hunting): picked mon.noname-a con 0x7f0a080a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a080a9d60 con 0x7f0a080a96b0 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0813ff10 con 0x7f0a08142a20 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a080aa1f0 con 0x7f0a080b4700 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0ed7b640 10 monclient(hunting): authenticate will time out at 5938.667958s 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0d318640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0a080a96b0 0x7f0a0813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33834/0 (socket says 172.21.15.111:33834) 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a07fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0a08142a20 0x7f0a0813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59090/0 (socket says 172.21.15.111:59090) 2024-03-25T17:39:03.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.532+0000 7f0a0d318640 1 -- 172.21.15.111:0/1380598329 learned_addr learned my addr 172.21.15.111:0/1380598329 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:03.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794316827 0 0) 0x7f0a0813ff10 con 0x7f0a08142a20 2024-03-25T17:39:03.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): my global_id is 4129 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f8003520 con 0x7f0a08142a20 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800992996 0 0) 0x7f0a080aa1f0 con 0x7f0a080b4700 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): my global_id is 4130 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0813ff10 con 0x7f0a080b4700 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052840525 0 0) 0x7f0a080a9d60 con 0x7f0a080a96b0 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 10 monclient(hunting): my global_id is 4134 2024-03-25T17:39:03.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.535+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a080aa1f0 con 0x7f0a080a96b0 2024-03-25T17:39:03.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3548970727 0 0) 0x7f09f8003520 con 0x7f0a08142a20 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 >> v1:172.21.15.44:6790/0 conn(0x7f0a080b4700 legacy=0x7f0a08142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 >> v1:172.21.15.44:6789/0 conn(0x7f0a080a96b0 legacy=0x7f0a0813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 1 monclient: found mon.noname-c 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a080b4ae0 con 0x7f0a08142a20 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09f40015c0 con 0x7f0a08142a20 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:03.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a077fe640 10 monclient: dump: 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a0ed7b640 5 monclient: authenticate success, global_id 4129 2024-03-25T17:39:03.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.538+0000 7f0a0ed7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.540+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f4001720 con 0x7f0a08142a20 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.540+0000 7f0a077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.540+0000 7f0a077fe640 1 -- 172.21.15.111:0/1380598329 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09f4001b30 con 0x7f0a08142a20 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.540+0000 7f0a077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.540+0000 7f0a077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.540+0000 7f0a077fe640 10 monclient: dump: 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.542 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.540+0000 7f0a0ed7b640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.540+0000 7f0a0ed7b640 4 set_mon_vals no callback set 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.541+0000 7f0a0ed7b640 10 monclient: shutdown 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.541+0000 7f0a0ed7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.541+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/1380598329 >> v1:172.21.15.111:6789/0 conn(0x7f0a08142a20 legacy=0x7f0a0813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.541+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/1380598329 shutdown_connections 2024-03-25T17:39:03.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.541+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/1380598329 >> 172.21.15.111:0/1380598329 conn(0x7f0a080668e0 msgr2=0x7f0a08066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:03.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.541+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/1380598329 shutdown_connections 2024-03-25T17:39:03.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.541+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/1380598329 wait complete. 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.542+0000 7f0a0ed7b640 10 monclient: build_initial_monmap 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.542+0000 7f0a0ed7b640 10 monclient: monmap: 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T17:39:03.546 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.542+0000 7f0a0ed7b640 1 librados: starting msgr at 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.542+0000 7f0a0ed7b640 1 librados: starting objecter 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 1 Processor -- start 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 1 -- start start 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 1 librados: setting wanted keys 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 1 librados: calling monclient init 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 10 monclient: init 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 10 monclient(hunting): picked mon.noname-c con 0x7f0a080a96b0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:03.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 10 monclient(hunting): picked mon.noname-a con 0x7f0a080b4700 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 10 monclient(hunting): picked mon.noname-b con 0x7f0a08142a20 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a080ac1b0 con 0x7f0a080b4700 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0813a640 con 0x7f0a08142a20 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0813c430 con 0x7f0a080a96b0 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.543+0000 7f0a0ed7b640 10 monclient(hunting): authenticate will time out at 5938.679497s 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a0cb17640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f0a080a96b0 0x7f0a0813bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:40922/0 (socket says 172.21.15.111:40922) 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a0d318640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0a08142a20 0x7f0a081e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59098/0 (socket says 172.21.15.111:59098) 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a0cb17640 1 -- 172.21.15.111:0/3639466907 learned_addr learned my addr 172.21.15.111:0/3639466907 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a057fa640 10 client.?.objecter ms_handle_connect 0x7f0a08142a20 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a057fa640 10 client.?.objecter ms_handle_connect 0x7f0a080b4700 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a057fa640 10 client.?.objecter ms_handle_connect 0x7f0a080a96b0 2024-03-25T17:39:03.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.544+0000 7f0a057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:03.549 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:03.546+0000 7f9ccb458040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T17:39:03.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.546+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668248382 0 0) 0x7f0a0813a640 con 0x7f0a08142a20 2024-03-25T17:39:03.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.546+0000 7f0a057fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.546+0000 7f0a057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.546+0000 7f0a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.546+0000 7f0a057fa640 10 monclient(hunting): my global_id is 4132 2024-03-25T17:39:03.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.546+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ec004eb0 con 0x7f0a08142a20 2024-03-25T17:39:03.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024961910 0 0) 0x7f0a080ac1b0 con 0x7f0a080b4700 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 10 monclient(hunting): my global_id is 4137 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0813a640 con 0x7f0a080b4700 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682198808 0 0) 0x7f0a0813c430 con 0x7f0a080a96b0 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 10 monclient(hunting): my global_id is 4133 2024-03-25T17:39:03.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.547+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a080ac1b0 con 0x7f0a080a96b0 2024-03-25T17:39:03.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.551+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2432611604 0 0) 0x7f09ec004eb0 con 0x7f0a08142a20 2024-03-25T17:39:03.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.551+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a0813c430 con 0x7f0a08142a20 2024-03-25T17:39:03.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.551+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0a00001bd0 con 0x7f0a08142a20 2024-03-25T17:39:03.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.551+0000 7f0a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.551+0000 7f0a057fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:03.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.551+0000 7f0a057fa640 10 monclient(hunting): dump: 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.551+0000 7f0a057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.552+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262272991 0 0) 0x7f0a0813a640 con 0x7f0a080b4700 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.552+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09ec004eb0 con 0x7f0a080b4700 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.552+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09f4001a30 con 0x7f0a080b4700 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.552+0000 7f0a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.552+0000 7f0a057fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.552+0000 7f0a057fa640 10 monclient(hunting): dump: 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.556 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1038868691 0 0) 0x7f0a080ac1b0 con 0x7f0a080a96b0 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a0813a640 con 0x7f0a080a96b0 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09fc001930 con 0x7f0a080a96b0 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 10 monclient(hunting): dump: 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.557 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1863883696 0 0) 0x7f0a0813c430 con 0x7f0a08142a20 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 >> v1:172.21.15.44:6790/0 conn(0x7f0a080a96b0 legacy=0x7f0a0813bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 >> v1:172.21.15.44:6789/0 conn(0x7f0a080b4700 legacy=0x7f0a08138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 10 monclient: _finish_hunting 0 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 1 monclient: found mon.b 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0813c5c0 con 0x7f0a08142a20 2024-03-25T17:39:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 10 monclient: _finish_auth 0 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 10 monclient: _check_auth_tickets 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 5 monclient: authenticate success, global_id 4132 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 10 monclient: _renew_subs 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0813b080 con 0x7f0a08142a20 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 10 monclient: _renew_subs 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a080ac720 con 0x7f0a08142a20 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.554+0000 7f0a0ed7b640 1 librados: init done 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.555+0000 7f0a0ed7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.555+0000 7f0a0ed7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.555+0000 7f0a0ed7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.555+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0813a7d0 con 0x7f0a08142a20 2024-03-25T17:39:03.595 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:03.595+0000 7f9ccb458040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T17:39:03.606 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:03.606+0000 7fca18a2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T17:39:03.653 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:03.653+0000 7fca18a2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T17:39:03.712 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:03.711+0000 7f9ccb458040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a00001fe0 con 0x7f0a08142a20 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0a00004d00 con 0x7f0a08142a20 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 10 monclient: dump: 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:03.750 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a06ffd640 4 set_mon_vals no callback set 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0a000052b0 con 0x7f0a08142a20 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2984241826 0 0) 0x7f0a000068a0 con 0x7f0a08142a20 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 10 client.4132.objecter ms_dispatch 0x7f0a08000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0a00018e00 con 0x7f0a08142a20 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:03.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.749+0000 7f0a057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:03.769 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:03.768+0000 7fca18a2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:39:03.810 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:03.810+0000 7f9ccb458040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T17:39:03.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.821+0000 7f0a0ed7b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ed0322fe-4a1a-4b8c-9886-9d7b9a18530e", "id": 3}] 2024-03-25T17:39:03.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.821+0000 7f0a0ed7b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ed0322fe-4a1a-4b8c-9886-9d7b9a18530e", "id": 3}] 2024-03-25T17:39:03.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.821+0000 7f0a0ed7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:03.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.821+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ed0322fe-4a1a-4b8c-9886-9d7b9a18530e", "id": 3} v 0) v1 -- 0x7f0a080aca20 con 0x7f0a08142a20 2024-03-25T17:39:03.867 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:03.867+0000 7fca18a2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T17:39:03.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.936+0000 7f0a057fa640 1 -- 172.21.15.111:0/3639466907 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ed0322fe-4a1a-4b8c-9886-9d7b9a18530e", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3895995538 0 525046306) 0x7f0a00006540 con 0x7f0a08142a20 2024-03-25T17:39:03.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.936+0000 7f0a057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ed0322fe-4a1a-4b8c-9886-9d7b9a18530e", "id": 3}] 2024-03-25T17:39:03.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.936+0000 7f0a057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:03.937 INFO:teuthology.orchestra.run.smithi111.stdout:3 2024-03-25T17:39:03.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 10 librados: watch_flush enter 2024-03-25T17:39:03.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 10 librados: watch_flush exit 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 10 client.4132.objecter successfully canceled tick 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 10 monclient: shutdown 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 >> v1:172.21.15.111:6789/0 conn(0x7f0a08142a20 legacy=0x7f0a081e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 shutdown_connections 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 >> 172.21.15.111:0/3639466907 conn(0x7f0a080668e0 msgr2=0x7f0a0813e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 shutdown_connections 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.937+0000 7f0a0ed7b640 1 -- 172.21.15.111:0/3639466907 wait complete. 2024-03-25T17:39:03.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:03.938+0000 7f0a0ed7b640 1 librados: shutdown 2024-03-25T17:39:03.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new af25a07f-955a-4cb4-b3b3-cdc2fbfad425 4 2024-03-25T17:39:03.977 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:03.976+0000 7f9ccb458040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:39:04.020 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:04.020+0000 7f9ccb458040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T17:39:04.030 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:04.030+0000 7fca18a2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:39:04.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.047+0000 7f6895dc8640 10 monclient: get_monmap_and_config 2024-03-25T17:39:04.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.047+0000 7f6895dc8640 10 monclient: build_initial_monmap 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.047+0000 7f6895dc8640 10 monclient: monmap: 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:39:04.048302+0000 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:39:04.048302+0000 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:04.048 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 1 Processor -- start 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 1 -- start start 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 10 monclient: init 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 10 monclient(hunting): picked mon.noname-c con 0x7f68900b4700 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 10 monclient(hunting): picked mon.noname-a con 0x7f6890142a20 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 10 monclient(hunting): picked mon.noname-b con 0x7f68900a96b0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68900a9d60 con 0x7f6890142a20 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689013ff10 con 0x7f68900b4700 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68900aa1f0 con 0x7f68900a96b0 2024-03-25T17:39:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.048+0000 7f6895dc8640 10 monclient(hunting): authenticate will time out at 5939.184412s 2024-03-25T17:39:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.049+0000 7f688effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6890142a20 0x7f689013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33852/0 (socket says 172.21.15.111:33852) 2024-03-25T17:39:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.049+0000 7f688effd640 1 -- 172.21.15.111:0/761137982 learned_addr learned my addr 172.21.15.111:0/761137982 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:04.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489545001 0 0) 0x7f689013ff10 con 0x7f68900b4700 2024-03-25T17:39:04.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 10 monclient(hunting): my global_id is 4135 2024-03-25T17:39:04.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6878003520 con 0x7f68900b4700 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828361688 0 0) 0x7f68900a9d60 con 0x7f6890142a20 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 10 monclient(hunting): my global_id is 4140 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.051+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f689013ff10 con 0x7f6890142a20 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.052+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443567925 0 0) 0x7f68900aa1f0 con 0x7f68900a96b0 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.052+0000 7f688e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.052+0000 7f688e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.052+0000 7f688e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.052+0000 7f688e7fc640 10 monclient(hunting): my global_id is 4136 2024-03-25T17:39:04.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.052+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68900a9d60 con 0x7f68900a96b0 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3994390546 0 0) 0x7f6878003520 con 0x7f68900b4700 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 >> v1:172.21.15.44:6790/0 conn(0x7f68900a96b0 legacy=0x7f689013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 >> v1:172.21.15.44:6789/0 conn(0x7f6890142a20 legacy=0x7f689013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 1 monclient: found mon.noname-c 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68900b4ae0 con 0x7f68900b4700 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:39:04.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6880001690 con 0x7f68900b4700 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f6895dc8640 5 monclient: authenticate success, global_id 4135 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f6895dc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f688e7fc640 10 monclient: dump: 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.057 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.058 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.058 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.055+0000 7f6895dc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:04.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.056+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68800017f0 con 0x7f68900b4700 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.056+0000 7f688e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.056+0000 7f688e7fc640 1 -- 172.21.15.111:0/761137982 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6880001c00 con 0x7f68900b4700 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.056+0000 7f688e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.056+0000 7f688e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.056+0000 7f688e7fc640 10 monclient: dump: 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.060 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.056+0000 7f6895dc8640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.056+0000 7f6895dc8640 4 set_mon_vals no callback set 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.057+0000 7f6895dc8640 10 monclient: shutdown 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.057+0000 7f6895dc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.057+0000 7f6895dc8640 1 -- 172.21.15.111:0/761137982 >> v1:172.21.15.111:6789/0 conn(0x7f68900b4700 legacy=0x7f6890142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.057+0000 7f6895dc8640 1 -- 172.21.15.111:0/761137982 shutdown_connections 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.057+0000 7f6895dc8640 1 -- 172.21.15.111:0/761137982 >> 172.21.15.111:0/761137982 conn(0x7f68900668e0 msgr2=0x7f6890066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.057+0000 7f6895dc8640 1 -- 172.21.15.111:0/761137982 shutdown_connections 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.057+0000 7f6895dc8640 1 -- 172.21.15.111:0/761137982 wait complete. 2024-03-25T17:39:04.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.058+0000 7f6895dc8640 10 monclient: build_initial_monmap 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.058+0000 7f6895dc8640 10 monclient: monmap: 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.059+0000 7f6895dc8640 1 librados: starting msgr at 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.059+0000 7f6895dc8640 1 librados: starting objecter 2024-03-25T17:39:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.059+0000 7f6895dc8640 1 Processor -- start 2024-03-25T17:39:04.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.059+0000 7f6895dc8640 1 -- start start 2024-03-25T17:39:04.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.059+0000 7f6895dc8640 1 librados: setting wanted keys 2024-03-25T17:39:04.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.059+0000 7f6895dc8640 1 librados: calling monclient init 2024-03-25T17:39:04.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.059+0000 7f6895dc8640 10 monclient: init 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 10 monclient(hunting): picked mon.noname-b con 0x7f68900a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 10 monclient(hunting): picked mon.noname-a con 0x7f68900b4700 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 10 monclient(hunting): picked mon.noname-c con 0x7f6890142a20 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6890139620 con 0x7f68900b4700 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689013a6a0 con 0x7f68900a96b0 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6890139890 con 0x7f6890142a20 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f6895dc8640 10 monclient(hunting): authenticate will time out at 5939.196264s 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f688effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f68900b4700 0x7f6890139f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33866/0 (socket says 172.21.15.111:33866) 2024-03-25T17:39:04.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.060+0000 7f688effd640 1 -- 172.21.15.111:0/1219629369 learned_addr learned my addr 172.21.15.111:0/1219629369 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.061+0000 7f6877fff640 10 client.?.objecter ms_handle_connect 0x7f68900b4700 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.061+0000 7f6877fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.061+0000 7f6877fff640 10 client.?.objecter ms_handle_connect 0x7f68900a96b0 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.061+0000 7f6877fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.061+0000 7f6877fff640 10 client.?.objecter ms_handle_connect 0x7f6890142a20 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.061+0000 7f6877fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.063+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221482553 0 0) 0x7f689013a6a0 con 0x7f68900a96b0 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.063+0000 7f6877fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.063+0000 7f6877fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.063+0000 7f6877fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.063+0000 7f6877fff640 10 monclient(hunting): my global_id is 4138 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.063+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f687c004eb0 con 0x7f68900a96b0 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.064+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348462230 0 0) 0x7f6890139890 con 0x7f6890142a20 2024-03-25T17:39:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.064+0000 7f6877fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.064+0000 7f6877fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.064+0000 7f6877fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.064+0000 7f6877fff640 10 monclient(hunting): my global_id is 4139 2024-03-25T17:39:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.064+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f689013a6a0 con 0x7f6890142a20 2024-03-25T17:39:04.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.067+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818952624 0 0) 0x7f687c004eb0 con 0x7f68900a96b0 2024-03-25T17:39:04.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.067+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6890139890 con 0x7f68900a96b0 2024-03-25T17:39:04.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.067+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6880001b00 con 0x7f68900a96b0 2024-03-25T17:39:04.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.067+0000 7f6877fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.067+0000 7f6877fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.067+0000 7f6877fff640 10 monclient(hunting): dump: 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.067+0000 7f6877fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:04.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.068+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915646409 0 0) 0x7f6890139620 con 0x7f68900b4700 2024-03-25T17:39:04.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.068+0000 7f6877fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.068+0000 7f6877fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.068+0000 7f6877fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.068+0000 7f6877fff640 10 monclient(hunting): my global_id is 4143 2024-03-25T17:39:04.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.068+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f687c004eb0 con 0x7f68900b4700 2024-03-25T17:39:04.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.069+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2112944150 0 0) 0x7f689013a6a0 con 0x7f6890142a20 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.069+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6890139620 con 0x7f6890142a20 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.069+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6888001b50 con 0x7f6890142a20 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.069+0000 7f6877fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.069+0000 7f6877fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.069+0000 7f6877fff640 10 monclient(hunting): dump: 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.071 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.072 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.077 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:04.077+0000 7fca18a2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T17:39:04.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.080+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150328830 0 0) 0x7f687c004eb0 con 0x7f68900b4700 2024-03-25T17:39:04.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.080+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f689013a6a0 con 0x7f68900b4700 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.080+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f68840018f0 con 0x7f68900b4700 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.080+0000 7f6877fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.080+0000 7f6877fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.080+0000 7f6877fff640 10 monclient(hunting): dump: 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.081 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1546917949 0 0) 0x7f6890139890 con 0x7f68900a96b0 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 >> v1:172.21.15.44:6790/0 conn(0x7f6890142a20 legacy=0x7f68901e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 >> v1:172.21.15.44:6789/0 conn(0x7f68900b4700 legacy=0x7f6890139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 10 monclient: _finish_hunting 0 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 1 monclient: found mon.b 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6890139b70 con 0x7f68900a96b0 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 10 monclient: _finish_auth 0 2024-03-25T17:39:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 10 monclient: _check_auth_tickets 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6877fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 5 monclient: authenticate success, global_id 4138 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 10 monclient: _renew_subs 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68901388c0 con 0x7f68900a96b0 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 10 monclient: _renew_subs 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6890135ab0 con 0x7f68900a96b0 2024-03-25T17:39:04.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.114+0000 7f6895dc8640 1 librados: init done 2024-03-25T17:39:04.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.115+0000 7f6895dc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.115+0000 7f6895dc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.115+0000 7f6895dc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.115+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689004f730 con 0x7f68900a96b0 2024-03-25T17:39:04.189 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:04.189+0000 7f9ccb458040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T17:39:04.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68800038a0 con 0x7f68900a96b0 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6880003bb0 con 0x7f68900a96b0 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 10 monclient: dump: 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.216 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f68800040a0 con 0x7f68900a96b0 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1054738420 0 0) 0x7f68800055d0 con 0x7f68900a96b0 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f688dffb640 4 set_mon_vals no callback set 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 10 client.4138.objecter ms_dispatch 0x7f6890000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T17:39:04.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6880005950 con 0x7f68900a96b0 2024-03-25T17:39:04.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.215+0000 7f6877fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:04.238 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:04.238+0000 7fca18a2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T17:39:04.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.288+0000 7f6895dc8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "af25a07f-955a-4cb4-b3b3-cdc2fbfad425", "id": 4}] 2024-03-25T17:39:04.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.288+0000 7f6895dc8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "af25a07f-955a-4cb4-b3b3-cdc2fbfad425", "id": 4}] 2024-03-25T17:39:04.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.288+0000 7f6895dc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.288+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "af25a07f-955a-4cb4-b3b3-cdc2fbfad425", "id": 4} v 0) v1 -- 0x7f689013e650 con 0x7f68900a96b0 2024-03-25T17:39:04.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.351+0000 7f6877fff640 1 -- 172.21.15.111:0/1219629369 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "af25a07f-955a-4cb4-b3b3-cdc2fbfad425", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3920091292 0 1697012583) 0x7f6880005190 con 0x7f68900a96b0 2024-03-25T17:39:04.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.351+0000 7f6877fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "af25a07f-955a-4cb4-b3b3-cdc2fbfad425", "id": 4}] 2024-03-25T17:39:04.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.351+0000 7f6877fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:04.352 INFO:teuthology.orchestra.run.smithi111.stdout:4 2024-03-25T17:39:04.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 10 librados: watch_flush enter 2024-03-25T17:39:04.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 10 librados: watch_flush exit 2024-03-25T17:39:04.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 10 client.4138.objecter successfully canceled tick 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 10 monclient: shutdown 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 >> v1:172.21.15.111:6789/0 conn(0x7f68900a96b0 legacy=0x7f6890137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 shutdown_connections 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 >> 172.21.15.111:0/1219629369 conn(0x7f68900668e0 msgr2=0x7f68900b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 shutdown_connections 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 1 -- 172.21.15.111:0/1219629369 wait complete. 2024-03-25T17:39:04.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.352+0000 7f6895dc8640 1 librados: shutdown 2024-03-25T17:39:04.363 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 7d1a4285-ebec-44ca-8893-ce7dfe83489c 5 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.462+0000 7fb42aec1640 10 monclient: get_monmap_and_config 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.462+0000 7fb42aec1640 10 monclient: build_initial_monmap 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.462+0000 7fb42aec1640 10 monclient: monmap: 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:39:04.463257+0000 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:39:04.463257+0000 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:04.464 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 1 Processor -- start 2024-03-25T17:39:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 1 -- start start 2024-03-25T17:39:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 10 monclient: init 2024-03-25T17:39:04.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:04.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:04.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 10 monclient(hunting): picked mon.noname-b con 0x7fb4240b4700 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:04.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 10 monclient(hunting): picked mon.noname-c con 0x7fb424142a20 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 10 monclient(hunting): picked mon.noname-a con 0x7fb4240a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4240a9d60 con 0x7fb4240a96b0 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42413ff10 con 0x7fb424142a20 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4240aa1f0 con 0x7fb4240b4700 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.463+0000 7fb42aec1640 10 monclient(hunting): authenticate will time out at 5939.599362s 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.464+0000 7fb428c5d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fb4240b4700 0x7fb424142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:40952/0 (socket says 172.21.15.111:40952) 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.464+0000 7fb428c5d640 1 -- 172.21.15.111:0/2819691793 learned_addr learned my addr 172.21.15.111:0/2819691793 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.464+0000 7fb42945e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb4240a96b0 0x7fb42413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33880/0 (socket says 172.21.15.111:33880) 2024-03-25T17:39:04.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704827724 0 0) 0x7fb42413ff10 con 0x7fb424142a20 2024-03-25T17:39:04.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 10 monclient(hunting): my global_id is 4141 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb414003520 con 0x7fb424142a20 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060097779 0 0) 0x7fb4240a9d60 con 0x7fb4240a96b0 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 10 monclient(hunting): my global_id is 4146 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.466+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb42413ff10 con 0x7fb4240a96b0 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.467+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861585341 0 0) 0x7fb4240aa1f0 con 0x7fb4240b4700 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.467+0000 7fb4237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.467+0000 7fb4237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.467+0000 7fb4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.467+0000 7fb4237fe640 10 monclient(hunting): my global_id is 4142 2024-03-25T17:39:04.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.467+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4240a9d60 con 0x7fb4240b4700 2024-03-25T17:39:04.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960388170 0 0) 0x7fb414003520 con 0x7fb424142a20 2024-03-25T17:39:04.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 >> v1:172.21.15.44:6790/0 conn(0x7fb4240b4700 legacy=0x7fb424142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 >> v1:172.21.15.44:6789/0 conn(0x7fb4240a96b0 legacy=0x7fb42413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:04.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 1 monclient: found mon.noname-c 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4240b4ae0 con 0x7fb424142a20 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb410001690 con 0x7fb424142a20 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb4237fe640 10 monclient: dump: 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb42aec1640 5 monclient: authenticate success, global_id 4141 2024-03-25T17:39:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.469+0000 7fb42aec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:04.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.471+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4100017f0 con 0x7fb424142a20 2024-03-25T17:39:04.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.471+0000 7fb4237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:04.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.471+0000 7fb4237fe640 1 -- 172.21.15.111:0/2819691793 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb410001c00 con 0x7fb424142a20 2024-03-25T17:39:04.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.471+0000 7fb4237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.471+0000 7fb4237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:04.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.471+0000 7fb4237fe640 10 monclient: dump: 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.471+0000 7fb42aec1640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.471+0000 7fb42aec1640 4 set_mon_vals no callback set 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.472+0000 7fb42aec1640 10 monclient: shutdown 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.472+0000 7fb42aec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.472+0000 7fb42aec1640 1 -- 172.21.15.111:0/2819691793 >> v1:172.21.15.111:6789/0 conn(0x7fb424142a20 legacy=0x7fb42413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.472+0000 7fb42aec1640 1 -- 172.21.15.111:0/2819691793 shutdown_connections 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.472+0000 7fb42aec1640 1 -- 172.21.15.111:0/2819691793 >> 172.21.15.111:0/2819691793 conn(0x7fb4240668e0 msgr2=0x7fb424066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.472+0000 7fb42aec1640 1 -- 172.21.15.111:0/2819691793 shutdown_connections 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.472+0000 7fb42aec1640 1 -- 172.21.15.111:0/2819691793 wait complete. 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 10 monclient: build_initial_monmap 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 10 monclient: monmap: 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 1 librados: starting msgr at 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 1 librados: starting objecter 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 1 Processor -- start 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 1 -- start start 2024-03-25T17:39:04.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 1 librados: setting wanted keys 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 1 librados: calling monclient init 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.473+0000 7fb42aec1640 10 monclient: init 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 10 monclient(hunting): picked mon.noname-a con 0x7fb4240a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 10 monclient(hunting): picked mon.noname-b con 0x7fb4240b4700 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 10 monclient(hunting): picked mon.noname-c con 0x7fb424142a20 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4241d7590 con 0x7fb4240a96b0 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42413ccc0 con 0x7fb4240b4700 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4241d8180 con 0x7fb424142a20 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42aec1640 10 monclient(hunting): authenticate will time out at 5939.609978s 2024-03-25T17:39:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb428c5d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb4240a96b0 0x7fb4241d7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33892/0 (socket says 172.21.15.111:33892) 2024-03-25T17:39:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb428c5d640 1 -- 172.21.15.111:0/3116468392 learned_addr learned my addr 172.21.15.111:0/3116468392 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb42945e640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fb424142a20 0x7fb42413b870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:40966/0 (socket says 172.21.15.111:40966) 2024-03-25T17:39:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.474+0000 7fb4217fa640 10 client.?.objecter ms_handle_connect 0x7fb4240a96b0 2024-03-25T17:39:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.475+0000 7fb4217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.475+0000 7fb4217fa640 10 client.?.objecter ms_handle_connect 0x7fb4240b4700 2024-03-25T17:39:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.475+0000 7fb4217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.475+0000 7fb4217fa640 10 client.?.objecter ms_handle_connect 0x7fb424142a20 2024-03-25T17:39:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.475+0000 7fb4217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.476+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193404710 0 0) 0x7fb42413ccc0 con 0x7fb4240b4700 2024-03-25T17:39:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.476+0000 7fb4217fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.476+0000 7fb4217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.476+0000 7fb4217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.476+0000 7fb4217fa640 10 monclient(hunting): my global_id is 4144 2024-03-25T17:39:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.476+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb408004eb0 con 0x7fb4240b4700 2024-03-25T17:39:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879321574 0 0) 0x7fb4241d7590 con 0x7fb4240a96b0 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 10 monclient(hunting): my global_id is 4149 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb42413ccc0 con 0x7fb4240a96b0 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103002534 0 0) 0x7fb4241d8180 con 0x7fb424142a20 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 10 monclient(hunting): my global_id is 4145 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.477+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4241d7590 con 0x7fb424142a20 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2121700370 0 0) 0x7fb408004eb0 con 0x7fb4240b4700 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4241d8180 con 0x7fb4240b4700 2024-03-25T17:39:04.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb410001b00 con 0x7fb4240b4700 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 10 monclient(hunting): dump: 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.484 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2072089264 0 0) 0x7fb42413ccc0 con 0x7fb4240a96b0 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb408004eb0 con 0x7fb4240a96b0 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb4180018f0 con 0x7fb4240a96b0 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.481+0000 7fb4217fa640 10 monclient(hunting): dump: 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.486 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1479730445 0 0) 0x7fb4241d8180 con 0x7fb4240b4700 2024-03-25T17:39:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 >> v1:172.21.15.44:6790/0 conn(0x7fb424142a20 legacy=0x7fb42413b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 >> v1:172.21.15.44:6789/0 conn(0x7fb4240a96b0 legacy=0x7fb4241d7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 10 monclient: _finish_hunting 0 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 1 monclient: found mon.b 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4241d8310 con 0x7fb4240b4700 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 10 monclient: _finish_auth 0 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 10 monclient: _check_auth_tickets 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb4217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb42aec1640 5 monclient: authenticate success, global_id 4144 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb42aec1640 10 monclient: _renew_subs 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb42aec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.484+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4241d3fd0 con 0x7fb4240b4700 2024-03-25T17:39:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 10 monclient: _renew_subs 2024-03-25T17:39:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4241d4f40 con 0x7fb4240b4700 2024-03-25T17:39:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 1 librados: init done 2024-03-25T17:39:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.485+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4241d73f0 con 0x7fb4240b4700 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4100038a0 con 0x7fb4240b4700 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb410003a10 con 0x7fb4240b4700 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 10 monclient: dump: 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb410003f00 con 0x7fb4240b4700 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb422ffd640 4 set_mon_vals no callback set 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2828483409 0 0) 0x7fb4100054a0 con 0x7fb4240b4700 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 10 client.4144.objecter ms_dispatch 0x7fb424000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T17:39:04.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T17:39:04.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T17:39:04.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T17:39:04.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb410005820 con 0x7fb4240b4700 2024-03-25T17:39:04.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.583+0000 7fb4217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:04.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.656+0000 7fb42aec1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7d1a4285-ebec-44ca-8893-ce7dfe83489c", "id": 5}] 2024-03-25T17:39:04.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.656+0000 7fb42aec1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7d1a4285-ebec-44ca-8893-ce7dfe83489c", "id": 5}] 2024-03-25T17:39:04.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.656+0000 7fb42aec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.656+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7d1a4285-ebec-44ca-8893-ce7dfe83489c", "id": 5} v 0) v1 -- 0x7fb42404f730 con 0x7fb4240b4700 2024-03-25T17:39:04.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.712+0000 7fb4217fa640 1 -- 172.21.15.111:0/3116468392 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7d1a4285-ebec-44ca-8893-ce7dfe83489c", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1194006927 0 1988412176) 0x7fb410004ff0 con 0x7fb4240b4700 2024-03-25T17:39:04.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.712+0000 7fb4217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7d1a4285-ebec-44ca-8893-ce7dfe83489c", "id": 5}] 2024-03-25T17:39:04.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.712+0000 7fb4217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:04.712 INFO:teuthology.orchestra.run.smithi111.stdout:5 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.712+0000 7fb42aec1640 10 librados: watch_flush enter 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.712+0000 7fb42aec1640 10 librados: watch_flush exit 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.712+0000 7fb42aec1640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.712+0000 7fb42aec1640 10 client.4144.objecter successfully canceled tick 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.713+0000 7fb42aec1640 10 monclient: shutdown 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.713+0000 7fb42aec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.713+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 >> v1:172.21.15.111:6789/0 conn(0x7fb4240b4700 legacy=0x7fb4241d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.713+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 shutdown_connections 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.713+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 >> 172.21.15.111:0/3116468392 conn(0x7fb4240668e0 msgr2=0x7fb4240b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.713+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 shutdown_connections 2024-03-25T17:39:04.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.713+0000 7fb42aec1640 1 -- 172.21.15.111:0/3116468392 wait complete. 2024-03-25T17:39:04.715 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.713+0000 7fb42aec1640 1 librados: shutdown 2024-03-25T17:39:04.724 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 0c5e74c5-4698-4e3f-91c8-5e5134445c37 6 2024-03-25T17:39:04.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.823+0000 7f5430481640 10 monclient: get_monmap_and_config 2024-03-25T17:39:04.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.823+0000 7f5430481640 10 monclient: build_initial_monmap 2024-03-25T17:39:04.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.823+0000 7f5430481640 10 monclient: monmap: 2024-03-25T17:39:04.824 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:04.824 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.824 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:39:04.824870+0000 2024-03-25T17:39:04.824 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:39:04.824870+0000 2024-03-25T17:39:04.825 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:04.825 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.825 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:04.825 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:04.825 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:04.825 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.824+0000 7f5430481640 1 Processor -- start 2024-03-25T17:39:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.824+0000 7f5430481640 1 -- start start 2024-03-25T17:39:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.824+0000 7f5430481640 10 monclient: init 2024-03-25T17:39:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 10 monclient(hunting): picked mon.noname-b con 0x7f54280b4700 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 10 monclient(hunting): picked mon.noname-c con 0x7f5428142a20 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 10 monclient(hunting): picked mon.noname-a con 0x7f54280a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:04.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280a9d60 con 0x7f54280a96b0 2024-03-25T17:39:04.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542813ff10 con 0x7f5428142a20 2024-03-25T17:39:04.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280aa1f0 con 0x7f54280b4700 2024-03-25T17:39:04.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:04.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f5430481640 10 monclient(hunting): authenticate will time out at 5939.961015s 2024-03-25T17:39:04.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f542e21d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f54280b4700 0x7f5428142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:40976/0 (socket says 172.21.15.111:40976) 2024-03-25T17:39:04.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.825+0000 7f542e21d640 1 -- 172.21.15.111:0/1937717565 learned_addr learned my addr 172.21.15.111:0/1937717565 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469816642 0 0) 0x7f542813ff10 con 0x7f5428142a20 2024-03-25T17:39:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): my global_id is 4147 2024-03-25T17:39:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5414003520 con 0x7f5428142a20 2024-03-25T17:39:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613410778 0 0) 0x7f54280aa1f0 con 0x7f54280b4700 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): my global_id is 4148 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542813ff10 con 0x7f54280b4700 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421372456 0 0) 0x7f54280a9d60 con 0x7f54280a96b0 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 10 monclient(hunting): my global_id is 4152 2024-03-25T17:39:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.828+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54280aa1f0 con 0x7f54280a96b0 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3069618226 0 0) 0x7f5414003520 con 0x7f5428142a20 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 >> v1:172.21.15.44:6790/0 conn(0x7f54280b4700 legacy=0x7f5428142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 >> v1:172.21.15.44:6789/0 conn(0x7f54280a96b0 legacy=0x7f542813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 10 monclient: _finish_hunting 0 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 1 monclient: found mon.noname-c 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54280b4ae0 con 0x7f5428142a20 2024-03-25T17:39:04.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 10 monclient: _finish_auth 0 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 10 monclient: _check_auth_tickets 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f541c001690 con 0x7f5428142a20 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f542d21b640 10 monclient: dump: 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.833 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.834 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.834 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.834 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.834 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f5430481640 5 monclient: authenticate success, global_id 4147 2024-03-25T17:39:04.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.831+0000 7f5430481640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f541c0017f0 con 0x7f5428142a20 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f542d21b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f542d21b640 1 -- 172.21.15.111:0/1937717565 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f541c001c40 con 0x7f5428142a20 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f542d21b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f542d21b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f542d21b640 10 monclient: dump: 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f5430481640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f5430481640 4 set_mon_vals no callback set 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f5430481640 10 monclient: shutdown 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f5430481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.833+0000 7f5430481640 1 -- 172.21.15.111:0/1937717565 >> v1:172.21.15.111:6789/0 conn(0x7f5428142a20 legacy=0x7f542813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.834+0000 7f5430481640 1 -- 172.21.15.111:0/1937717565 shutdown_connections 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.834+0000 7f5430481640 1 -- 172.21.15.111:0/1937717565 >> 172.21.15.111:0/1937717565 conn(0x7f54280668e0 msgr2=0x7f5428066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.834+0000 7f5430481640 1 -- 172.21.15.111:0/1937717565 shutdown_connections 2024-03-25T17:39:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.834+0000 7f5430481640 1 -- 172.21.15.111:0/1937717565 wait complete. 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.834+0000 7f5430481640 10 monclient: build_initial_monmap 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.834+0000 7f5430481640 10 monclient: monmap: 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 librados: starting msgr at 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 librados: starting objecter 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 Processor -- start 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 -- start start 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 librados: setting wanted keys 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 librados: calling monclient init 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 10 monclient: init 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 10 monclient(hunting): picked mon.noname-b con 0x7f54280a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 10 monclient(hunting): picked mon.noname-c con 0x7f54280b4700 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 10 monclient(hunting): picked mon.noname-a con 0x7f5428142a20 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54281d7320 con 0x7f5428142a20 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542812b3a0 con 0x7f54280a96b0 2024-03-25T17:39:04.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542812b530 con 0x7f54280b4700 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.835+0000 7f5430481640 10 monclient(hunting): authenticate will time out at 5939.971502s 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.836+0000 7f542e21d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f54280a96b0 0x7f542812ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59154/0 (socket says 172.21.15.111:59154) 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.836+0000 7f542e21d640 1 -- 172.21.15.111:0/1703989675 learned_addr learned my addr 172.21.15.111:0/1703989675 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.836+0000 7f541affd640 10 client.?.objecter ms_handle_connect 0x7f54280a96b0 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.836+0000 7f541affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.836+0000 7f541affd640 10 client.?.objecter ms_handle_connect 0x7f54280b4700 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.836+0000 7f541affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.836+0000 7f541affd640 10 client.?.objecter ms_handle_connect 0x7f5428142a20 2024-03-25T17:39:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.836+0000 7f541affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:04.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561434117 0 0) 0x7f542812b530 con 0x7f54280b4700 2024-03-25T17:39:04.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 10 monclient(hunting): my global_id is 4151 2024-03-25T17:39:04.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c004eb0 con 0x7f54280b4700 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634730890 0 0) 0x7f542812b3a0 con 0x7f54280a96b0 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 10 monclient(hunting): my global_id is 4150 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.838+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542812b530 con 0x7f54280a96b0 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.839+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583473278 0 0) 0x7f54281d7320 con 0x7f5428142a20 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.839+0000 7f541affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.839+0000 7f541affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.839+0000 7f541affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.839+0000 7f541affd640 10 monclient(hunting): my global_id is 4155 2024-03-25T17:39:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.839+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542812b3a0 con 0x7f5428142a20 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1475655409 0 0) 0x7f542812b530 con 0x7f54280a96b0 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54281d7320 con 0x7f54280a96b0 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5420001dd0 con 0x7f54280a96b0 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): dump: 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350025260 0 0) 0x7f542812b3a0 con 0x7f5428142a20 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f542812b530 con 0x7f5428142a20 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5424001850 con 0x7f5428142a20 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): dump: 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3673753450 0 0) 0x7f540c004eb0 con 0x7f54280b4700 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f542812b3a0 con 0x7f54280b4700 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f541c001b00 con 0x7f54280b4700 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:04.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.844+0000 7f541affd640 10 monclient(hunting): dump: 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.850 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3670950051 0 0) 0x7f54281d7320 con 0x7f54280a96b0 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 >> v1:172.21.15.44:6790/0 conn(0x7f54280b4700 legacy=0x7f54281d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 >> v1:172.21.15.44:6789/0 conn(0x7f5428142a20 legacy=0x7f54281e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 1 monclient: found mon.b 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542812cc10 con 0x7f54280a96b0 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 10 monclient: _finish_auth 0 2024-03-25T17:39:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f541affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 5 monclient: authenticate success, global_id 4150 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 10 monclient: _renew_subs 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54281d7f10 con 0x7f54280a96b0 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 10 monclient: _renew_subs 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5428132190 con 0x7f54280a96b0 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.847+0000 7f5430481640 1 librados: init done 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.848+0000 7f5430481640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.848+0000 7f5430481640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.848+0000 7f5430481640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.848+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542812ce60 con 0x7f54280a96b0 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5420004b60 con 0x7f54280a96b0 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5420004cd0 con 0x7f54280a96b0 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 10 monclient: dump: 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5420005240 con 0x7f54280a96b0 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f542ca1a640 4 set_mon_vals no callback set 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 455371620 0 0) 0x7f5420006ab0 con 0x7f54280a96b0 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 10 client.4150.objecter ms_dispatch 0x7f5428000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-25T17:39:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.852+0000 7f541affd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T17:39:04.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.948+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f54200065d0 con 0x7f54280a96b0 2024-03-25T17:39:04.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.948+0000 7f541affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:04.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:04.948+0000 7f541affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:05.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.023+0000 7f5430481640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0c5e74c5-4698-4e3f-91c8-5e5134445c37", "id": 6}] 2024-03-25T17:39:05.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.023+0000 7f5430481640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0c5e74c5-4698-4e3f-91c8-5e5134445c37", "id": 6}] 2024-03-25T17:39:05.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.023+0000 7f5430481640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:05.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.023+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0c5e74c5-4698-4e3f-91c8-5e5134445c37", "id": 6} v 0) v1 -- 0x7f542804f730 con 0x7f54280a96b0 2024-03-25T17:39:05.037 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:05.037+0000 7f9ccb458040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T17:39:05.079 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:05.078+0000 7fca18a2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T17:39:05.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.083+0000 7f541affd640 1 -- 172.21.15.111:0/1703989675 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0c5e74c5-4698-4e3f-91c8-5e5134445c37", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1851722905 0 1113811849) 0x7f5420006840 con 0x7f54280a96b0 2024-03-25T17:39:05.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.083+0000 7f541affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0c5e74c5-4698-4e3f-91c8-5e5134445c37", "id": 6}] 2024-03-25T17:39:05.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.083+0000 7f541affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:05.083 INFO:teuthology.orchestra.run.smithi111.stdout:6 2024-03-25T17:39:05.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.083+0000 7f5430481640 10 librados: watch_flush enter 2024-03-25T17:39:05.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 10 librados: watch_flush exit 2024-03-25T17:39:05.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T17:39:05.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 10 client.4150.objecter successfully canceled tick 2024-03-25T17:39:05.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 10 monclient: shutdown 2024-03-25T17:39:05.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:05.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 >> v1:172.21.15.111:6789/0 conn(0x7f54280a96b0 legacy=0x7f542812ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 shutdown_connections 2024-03-25T17:39:05.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 >> 172.21.15.111:0/1703989675 conn(0x7f54280668e0 msgr2=0x7f54280b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:05.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 shutdown_connections 2024-03-25T17:39:05.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 1 -- 172.21.15.111:0/1703989675 wait complete. 2024-03-25T17:39:05.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.084+0000 7f5430481640 1 librados: shutdown 2024-03-25T17:39:05.095 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new b36f57af-8e3c-45a4-a735-476c838df28e 7 2024-03-25T17:39:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.197+0000 7f6428904640 10 monclient: get_monmap_and_config 2024-03-25T17:39:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.197+0000 7f6428904640 10 monclient: build_initial_monmap 2024-03-25T17:39:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.197+0000 7f6428904640 10 monclient: monmap: 2024-03-25T17:39:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:39:05.198156+0000 2024-03-25T17:39:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:39:05.198156+0000 2024-03-25T17:39:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:05.199 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.197+0000 7f6428904640 1 Processor -- start 2024-03-25T17:39:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.197+0000 7f6428904640 1 -- start start 2024-03-25T17:39:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.197+0000 7f6428904640 10 monclient: init 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 10 monclient(hunting): picked mon.noname-b con 0x7f64240b4700 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 10 monclient(hunting): picked mon.noname-c con 0x7f6424142a20 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 10 monclient(hunting): picked mon.noname-a con 0x7f64240a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64240a9d60 con 0x7f64240a96b0 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642413ff10 con 0x7f6424142a20 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64240aa1f0 con 0x7f64240b4700 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.198+0000 7f6428904640 10 monclient(hunting): authenticate will time out at 5940.334254s 2024-03-25T17:39:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.199+0000 7f6422d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f64240a96b0 0x7f642413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33922/0 (socket says 172.21.15.111:33922) 2024-03-25T17:39:05.202 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.199+0000 7f6422d9d640 1 -- 172.21.15.111:0/3766962896 learned_addr learned my addr 172.21.15.111:0/3766962896 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329694820 0 0) 0x7f64240a9d60 con 0x7f64240a96b0 2024-03-25T17:39:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 10 monclient(hunting): my global_id is 4158 2024-03-25T17:39:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f640c003520 con 0x7f64240a96b0 2024-03-25T17:39:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100557829 0 0) 0x7f64240aa1f0 con 0x7f64240b4700 2024-03-25T17:39:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 10 monclient(hunting): my global_id is 4154 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.201+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64240a9d60 con 0x7f64240b4700 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.202+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841676673 0 0) 0x7f642413ff10 con 0x7f6424142a20 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.202+0000 7f642159a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.202+0000 7f642159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.202+0000 7f642159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.202+0000 7f642159a640 10 monclient(hunting): my global_id is 4153 2024-03-25T17:39:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.202+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64240aa1f0 con 0x7f6424142a20 2024-03-25T17:39:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3579592286 0 0) 0x7f640c003520 con 0x7f64240a96b0 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 >> v1:172.21.15.44:6790/0 conn(0x7f64240b4700 legacy=0x7f6424142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 >> v1:172.21.15.111:6789/0 conn(0x7f6424142a20 legacy=0x7f642413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 10 monclient: _finish_hunting 0 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 1 monclient: found mon.noname-a 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64240b4ae0 con 0x7f64240a96b0 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 10 monclient: _finish_auth 0 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 10 monclient: _check_auth_tickets 2024-03-25T17:39:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f641c001a10 con 0x7f64240a96b0 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.204+0000 7f642159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.205+0000 7f642159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.205+0000 7f642159a640 10 monclient: dump: 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.208 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.208 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.208 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:05.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.205+0000 7f6428904640 5 monclient: authenticate success, global_id 4158 2024-03-25T17:39:05.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.205+0000 7f6428904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.206+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f641c001b70 con 0x7f64240a96b0 2024-03-25T17:39:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.206+0000 7f642159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.206+0000 7f642159a640 1 -- 172.21.15.111:0/3766962896 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f641c001f80 con 0x7f64240a96b0 2024-03-25T17:39:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.206+0000 7f6428904640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.206+0000 7f6428904640 4 set_mon_vals no callback set 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.206+0000 7f642159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.206+0000 7f642159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.206+0000 7f642159a640 10 monclient: dump: 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.207+0000 7f6428904640 10 monclient: shutdown 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.207+0000 7f6428904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.207+0000 7f6428904640 1 -- 172.21.15.111:0/3766962896 >> v1:172.21.15.44:6789/0 conn(0x7f64240a96b0 legacy=0x7f642413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.207+0000 7f6428904640 1 -- 172.21.15.111:0/3766962896 shutdown_connections 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.207+0000 7f6428904640 1 -- 172.21.15.111:0/3766962896 >> 172.21.15.111:0/3766962896 conn(0x7f64240668e0 msgr2=0x7f6424066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.207+0000 7f6428904640 1 -- 172.21.15.111:0/3766962896 shutdown_connections 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.207+0000 7f6428904640 1 -- 172.21.15.111:0/3766962896 wait complete. 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 10 monclient: build_initial_monmap 2024-03-25T17:39:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 10 monclient: monmap: 2024-03-25T17:39:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T17:39:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T17:39:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T17:39:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T17:39:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 1 librados: starting msgr at 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 1 librados: starting objecter 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 1 Processor -- start 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 1 -- start start 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 1 librados: setting wanted keys 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 1 librados: calling monclient init 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.208+0000 7f6428904640 10 monclient: init 2024-03-25T17:39:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 10 monclient(hunting): picked mon.noname-a con 0x7f64240a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 10 monclient(hunting): picked mon.noname-c con 0x7f64240b4700 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 10 monclient(hunting): picked mon.noname-b con 0x7f6424142a20 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642413cb40 con 0x7f64240a96b0 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64240abf50 con 0x7f6424142a20 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64240ac1c0 con 0x7f64240b4700 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6428904640 10 monclient(hunting): authenticate will time out at 5940.344910s 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f642259c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f64240a96b0 0x7f642413c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:33924/0 (socket says 172.21.15.111:33924) 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f642259c640 1 -- 172.21.15.111:0/1420671197 learned_addr learned my addr 172.21.15.111:0/1420671197 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T17:39:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6412ffd640 10 client.?.objecter ms_handle_connect 0x7f64240a96b0 2024-03-25T17:39:05.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.209+0000 7f6412ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:05.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.210+0000 7f6412ffd640 10 client.?.objecter ms_handle_connect 0x7f64240b4700 2024-03-25T17:39:05.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.210+0000 7f6412ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:05.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.210+0000 7f6412ffd640 10 client.?.objecter ms_handle_connect 0x7f6424142a20 2024-03-25T17:39:05.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.210+0000 7f6412ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:05.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785159400 0 0) 0x7f642413cb40 con 0x7f64240a96b0 2024-03-25T17:39:05.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): my global_id is 4161 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6404004eb0 con 0x7f64240a96b0 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145570127 0 0) 0x7f64240abf50 con 0x7f6424142a20 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): my global_id is 4156 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f642413cb40 con 0x7f6424142a20 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532244657 0 0) 0x7f64240ac1c0 con 0x7f64240b4700 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 10 monclient(hunting): my global_id is 4157 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.212+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64240abf50 con 0x7f64240b4700 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.216+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3684607765 0 0) 0x7f6404004eb0 con 0x7f64240a96b0 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.216+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64240ac1c0 con 0x7f64240a96b0 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.216+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6414001a30 con 0x7f64240a96b0 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.216+0000 7f6412ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.216+0000 7f6412ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.216+0000 7f6412ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.222 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.216+0000 7f6412ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.217+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1873809382 0 0) 0x7f642413cb40 con 0x7f6424142a20 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.217+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6404004eb0 con 0x7f6424142a20 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.217+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f641c001e80 con 0x7f6424142a20 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.217+0000 7f6412ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.217+0000 7f6412ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.217+0000 7f6412ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.224 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.218+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4058583646 0 0) 0x7f64240abf50 con 0x7f64240b4700 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.218+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f642413cb40 con 0x7f64240b4700 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.218+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6418001790 con 0x7f64240b4700 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.218+0000 7f6412ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.218+0000 7f6412ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.218+0000 7f6412ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:05.225 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.219+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2744095746 0 0) 0x7f6404004eb0 con 0x7f6424142a20 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.219+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 >> v1:172.21.15.44:6790/0 conn(0x7f64240b4700 legacy=0x7f64241331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.219+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 >> v1:172.21.15.44:6789/0 conn(0x7f64240a96b0 legacy=0x7f642413c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.219+0000 7f6412ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6412ffd640 1 monclient: found mon.b 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6412ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:05.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6412ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642413b740 con 0x7f6424142a20 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6412ffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6412ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6412ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 5 monclient: authenticate success, global_id 4156 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 10 monclient: _renew_subs 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642413b930 con 0x7f6424142a20 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 10 monclient: _renew_subs 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:05.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64241e3850 con 0x7f6424142a20 2024-03-25T17:39:05.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.220+0000 7f6428904640 1 librados: init done 2024-03-25T17:39:05.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.221+0000 7f6428904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:05.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.221+0000 7f6428904640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:05.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.221+0000 7f6428904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:05.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.221+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64240aab00 con 0x7f6424142a20 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f641c001fe0 con 0x7f6424142a20 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f641c004fb0 con 0x7f6424142a20 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 10 monclient: dump: 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.321 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f641c0054a0 con 0x7f6424142a20 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6420d99640 4 set_mon_vals no callback set 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 4152124819 0 0) 0x7f641c006c40 con 0x7f6424142a20 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 10 client.4156.objecter ms_dispatch 0x7f6424000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T17:39:05.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T17:39:05.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T17:39:05.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f641c018dc0 con 0x7f6424142a20 2024-03-25T17:39:05.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:05.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.320+0000 7f6412ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:05.336 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:05.336+0000 7f9ccb458040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T17:39:05.373 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:05.372+0000 7fca18a2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T17:39:05.389 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:05.389+0000 7f9ccb458040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T17:39:05.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.392+0000 7f6428904640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b36f57af-8e3c-45a4-a735-476c838df28e", "id": 7}] 2024-03-25T17:39:05.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.392+0000 7f6428904640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b36f57af-8e3c-45a4-a735-476c838df28e", "id": 7}] 2024-03-25T17:39:05.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.392+0000 7f6428904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:05.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.392+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b36f57af-8e3c-45a4-a735-476c838df28e", "id": 7} v 0) v1 -- 0x7f64240b3670 con 0x7f6424142a20 2024-03-25T17:39:05.426 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:05.426+0000 7fca18a2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T17:39:05.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.459+0000 7f6412ffd640 1 -- 172.21.15.111:0/1420671197 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b36f57af-8e3c-45a4-a735-476c838df28e", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1789610381 0 1371665406) 0x7f641c0065e0 con 0x7f6424142a20 2024-03-25T17:39:05.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.459+0000 7f6412ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b36f57af-8e3c-45a4-a735-476c838df28e", "id": 7}] 2024-03-25T17:39:05.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.459+0000 7f6412ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:05.460 INFO:teuthology.orchestra.run.smithi111.stdout:7 2024-03-25T17:39:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 10 librados: watch_flush enter 2024-03-25T17:39:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 10 librados: watch_flush exit 2024-03-25T17:39:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T17:39:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 10 client.4156.objecter successfully canceled tick 2024-03-25T17:39:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 10 monclient: shutdown 2024-03-25T17:39:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 >> v1:172.21.15.111:6789/0 conn(0x7f6424142a20 legacy=0x7f64241338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 shutdown_connections 2024-03-25T17:39:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 >> 172.21.15.111:0/1420671197 conn(0x7f64240668e0 msgr2=0x7f642413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 shutdown_connections 2024-03-25T17:39:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 1 -- 172.21.15.111:0/1420671197 wait complete. 2024-03-25T17:39:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T17:39:05.460+0000 7f6428904640 1 librados: shutdown 2024-03-25T17:39:05.471 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:39:05.471 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T17:39:05.471 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T17:39:05.474 INFO:tasks.ceph.osd.0:Started 2024-03-25T17:39:05.474 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:39:05.474 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T17:39:05.474 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T17:39:05.477 INFO:tasks.ceph.osd.1:Started 2024-03-25T17:39:05.477 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:39:05.477 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T17:39:05.477 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T17:39:05.480 INFO:tasks.ceph.osd.2:Started 2024-03-25T17:39:05.480 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:39:05.480 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T17:39:05.480 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T17:39:05.483 INFO:tasks.ceph.osd.3:Started 2024-03-25T17:39:05.483 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:39:05.483 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T17:39:05.483 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T17:39:05.486 INFO:tasks.ceph.osd.4:Started 2024-03-25T17:39:05.486 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:39:05.486 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T17:39:05.486 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T17:39:05.489 INFO:tasks.ceph.osd.5:Started 2024-03-25T17:39:05.489 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:39:05.489 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T17:39:05.489 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T17:39:05.491 INFO:tasks.ceph.osd.6:Started 2024-03-25T17:39:05.491 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:39:05.492 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T17:39:05.492 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T17:39:05.495 INFO:tasks.ceph.osd.7:Started 2024-03-25T17:39:05.495 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:05.533 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:05.533+0000 7f9ccb458040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T17:39:05.559 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T17:39:05.558+0000 7fca18a2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.617+0000 7f9ce721e640 10 monclient: get_monmap_and_config 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.617+0000 7f9ce721e640 10 monclient: build_initial_monmap 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.617+0000 7f9ce721e640 10 monclient: monmap: 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 0 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:39:05.618291+0000 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:39:05.618291+0000 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:05.618 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:05.619 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:05.619 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:05.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.618+0000 7f9ce721e640 1 Processor -- start 2024-03-25T17:39:05.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.618+0000 7f9ce721e640 1 -- start start 2024-03-25T17:39:05.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.618+0000 7f9ce721e640 10 monclient: init 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.618+0000 7f9ce721e640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.618+0000 7f9ce721e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.618+0000 7f9ce721e640 10 monclient(hunting): picked mon.noname-a con 0x7f9ce00a9920 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.618+0000 7f9ce721e640 10 monclient(hunting): picked mon.noname-c con 0x7f9ce00b45a0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.619+0000 7f9ce721e640 10 monclient(hunting): picked mon.noname-b con 0x7f9ce0146450 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.619+0000 7f9ce721e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce006b100 con 0x7f9ce00a9920 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.619+0000 7f9ce721e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce00b2940 con 0x7f9ce00b45a0 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.619+0000 7f9ce721e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce013fec0 con 0x7f9ce0146450 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.619+0000 7f9ce721e640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.619+0000 7f9ce721e640 10 monclient(hunting): authenticate will time out at 5948.847138s 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.619+0000 7f9ce57bb640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f9ce0146450 0x7f9ce014dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:52430/0 (socket says 172.21.15.44:52430) 2024-03-25T17:39:05.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.619+0000 7f9ce57bb640 1 -- 172.21.15.44:0/1916619491 learned_addr learned my addr 172.21.15.44:0/1916619491 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:05.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.622+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178158609 0 0) 0x7f9ce013fec0 con 0x7f9ce0146450 2024-03-25T17:39:05.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.622+0000 7f9cdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.622+0000 7f9cdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.622+0000 7f9cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.622+0000 7f9cdf7fe640 10 monclient(hunting): my global_id is 4160 2024-03-25T17:39:05.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.622+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd0003520 con 0x7f9ce0146450 2024-03-25T17:39:05.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.623+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744133098 0 0) 0x7f9ce00b2940 con 0x7f9ce00b45a0 2024-03-25T17:39:05.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.623+0000 7f9cdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.623+0000 7f9cdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.623+0000 7f9cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.623+0000 7f9cdf7fe640 10 monclient(hunting): my global_id is 4159 2024-03-25T17:39:05.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.623+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ce013fec0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1916934669 0 0) 0x7f9cd0003520 con 0x7f9ce0146450 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 >> v1:172.21.15.111:6789/0 conn(0x7f9ce00b45a0 legacy=0x7f9ce0145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 >> v1:172.21.15.44:6789/0 conn(0x7f9ce00a9920 legacy=0x7f9ce013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 1 monclient: found mon.noname-b 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce0140050 con 0x7f9ce0146450 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:05.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9cd8001a10 con 0x7f9ce0146450 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9cdf7fe640 10 monclient: dump: 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.630 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9ce721e640 5 monclient: authenticate success, global_id 4160 2024-03-25T17:39:05.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.627+0000 7f9ce721e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.666+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cd8001b70 con 0x7f9ce0146450 2024-03-25T17:39:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.666+0000 7f9cdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.666+0000 7f9cdf7fe640 1 -- 172.21.15.44:0/1916619491 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9cd8001de0 con 0x7f9ce0146450 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.666+0000 7f9cdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.666+0000 7f9cdf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.666+0000 7f9cdf7fe640 10 monclient: dump: 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.668 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.666+0000 7f9ce721e640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.666+0000 7f9ce721e640 4 set_mon_vals no callback set 2024-03-25T17:39:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.667+0000 7f9ce721e640 10 monclient: shutdown 2024-03-25T17:39:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.667+0000 7f9ce721e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.667+0000 7f9ce721e640 1 -- 172.21.15.44:0/1916619491 >> v1:172.21.15.44:6790/0 conn(0x7f9ce0146450 legacy=0x7f9ce014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.668+0000 7f9ce721e640 1 -- 172.21.15.44:0/1916619491 shutdown_connections 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.668+0000 7f9ce721e640 1 -- 172.21.15.44:0/1916619491 >> 172.21.15.44:0/1916619491 conn(0x7f9ce00b7f40 msgr2=0x7f9ce00b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.668+0000 7f9ce721e640 1 -- 172.21.15.44:0/1916619491 shutdown_connections 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.668+0000 7f9ce721e640 1 -- 172.21.15.44:0/1916619491 wait complete. 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.669+0000 7f9ce721e640 10 monclient: build_initial_monmap 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.669+0000 7f9ce721e640 10 monclient: monmap: 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 0 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:created 0.000000 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.669+0000 7f9ce721e640 1 librados: starting msgr at 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.669+0000 7f9ce721e640 1 librados: starting objecter 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.669+0000 7f9ce721e640 1 Processor -- start 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.670+0000 7f9ce721e640 1 -- start start 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.670+0000 7f9ce721e640 1 librados: setting wanted keys 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.670+0000 7f9ce721e640 1 librados: calling monclient init 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.670+0000 7f9ce721e640 10 monclient: init 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.670+0000 7f9ce721e640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.670+0000 7f9ce721e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.670+0000 7f9ce721e640 10 monclient(hunting): picked mon.noname-b con 0x7f9ce00a9920 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce721e640 10 monclient(hunting): picked mon.noname-a con 0x7f9ce00b45a0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce721e640 10 monclient(hunting): picked mon.noname-c con 0x7f9ce0146450 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce721e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce00ac520 con 0x7f9ce00b45a0 2024-03-25T17:39:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce721e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce0144320 con 0x7f9ce00a9920 2024-03-25T17:39:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce721e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce0141340 con 0x7f9ce0146450 2024-03-25T17:39:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce721e640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce721e640 10 monclient(hunting): authenticate will time out at 5948.898868s 2024-03-25T17:39:05.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce57bb640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f9ce0146450 0x7f9ce01370e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:52444/0 (socket says 172.21.15.44:52444) 2024-03-25T17:39:05.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce4fba640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9ce00a9920 0x7f9ce0143c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:33894/0 (socket says 172.21.15.44:33894) 2024-03-25T17:39:05.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.671+0000 7f9ce57bb640 1 -- 172.21.15.44:0/4107819650 learned_addr learned my addr 172.21.15.44:0/4107819650 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:05.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.672+0000 7f9cdd7fa640 10 client.?.objecter ms_handle_connect 0x7f9ce00a9920 2024-03-25T17:39:05.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.672+0000 7f9cdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:05.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.672+0000 7f9cdd7fa640 10 client.?.objecter ms_handle_connect 0x7f9ce0146450 2024-03-25T17:39:05.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.672+0000 7f9cdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603145160 0 0) 0x7f9ce0144320 con 0x7f9ce00a9920 2024-03-25T17:39:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 10 monclient(hunting): my global_id is 4162 2024-03-25T17:39:05.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cc4004eb0 con 0x7f9ce00a9920 2024-03-25T17:39:05.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215031518 0 0) 0x7f9ce0141340 con 0x7f9ce0146450 2024-03-25T17:39:05.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 10 monclient(hunting): my global_id is 4163 2024-03-25T17:39:05.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.675+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ce0144320 con 0x7f9ce0146450 2024-03-25T17:39:05.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.678+0000 7f9cdd7fa640 10 client.?.objecter ms_handle_connect 0x7f9ce00b45a0 2024-03-25T17:39:05.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.678+0000 7f9cdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.680+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371024022 0 0) 0x7f9ce00ac520 con 0x7f9ce00b45a0 2024-03-25T17:39:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.680+0000 7f9cdd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.680+0000 7f9cdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.680+0000 7f9cdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.680+0000 7f9cdd7fa640 10 monclient(hunting): my global_id is 4164 2024-03-25T17:39:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.680+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ce0141340 con 0x7f9ce00b45a0 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.682+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2169399796 0 0) 0x7f9ce0144320 con 0x7f9ce0146450 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.682+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ce00ac520 con 0x7f9ce0146450 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.682+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9cd8001f40 con 0x7f9ce0146450 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.682+0000 7f9cdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.682+0000 7f9cdd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.682+0000 7f9cdd7fa640 10 monclient(hunting): dump: 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.682+0000 7f9cdd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.682+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3777391546 0 0) 0x7f9cc4004eb0 con 0x7f9ce00a9920 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.683+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ce0144320 con 0x7f9ce00a9920 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.683+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9cd4001820 con 0x7f9ce00a9920 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.683+0000 7f9cdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.683+0000 7f9cdd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.683+0000 7f9cdd7fa640 10 monclient(hunting): dump: 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.686 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.687+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 863533185 0 0) 0x7f9ce0141340 con 0x7f9ce00b45a0 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.688+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cc4004eb0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.688+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9ccc0018f0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.688+0000 7f9cdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.688+0000 7f9cdd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.688+0000 7f9cdd7fa640 10 monclient(hunting): dump: 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.689 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2335530354 0 0) 0x7f9cc4004eb0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 >> v1:172.21.15.44:6790/0 conn(0x7f9ce0146450 legacy=0x7f9ce01370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 >> v1:172.21.15.111:6789/0 conn(0x7f9ce00a9920 legacy=0x7f9ce0143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 1 monclient: found mon.a 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce01d81d0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 10 monclient: _finish_auth 0 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9cdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 5 monclient: authenticate success, global_id 4164 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 10 monclient: _renew_subs 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce01d8480 con 0x7f9ce00b45a0 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 10 client.4164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 10 monclient: _renew_subs 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce01379d0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.692+0000 7f9ce721e640 1 librados: init done 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.694+0000 7f9ce721e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.694+0000 7f9ce721e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.694+0000 7f9ce721e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.694+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce004f730 con 0x7f9ce00b45a0 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ccc0035f0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9ccc003900 con 0x7f9ce00b45a0 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 10 monclient: dump: 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9ccc003df0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdeffd640 4 set_mon_vals no callback set 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f9ccc005620 con 0x7f9ce00b45a0 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 10 client.4164.objecter ms_dispatch 0x7f9ce0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 3 client.4164.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 3 client.4164.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 20 client.4164.objecter dump_active .. 0 homeless 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9ccc005a00 con 0x7f9ce00b45a0 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:05.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.799+0000 7f9cdd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:05.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.874+0000 7f9ce721e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:05.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.874+0000 7f9ce721e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:05.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.874+0000 7f9ce721e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:05.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.874+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ce013b4d0 con 0x7f9ce00b45a0 2024-03-25T17:39:05.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:05.937+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3369074659 0 0) 0x7f9ccc004f50 con 0x7f9ce00b45a0 2024-03-25T17:39:06.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.096+0000 7f9cdd7fa640 1 -- 172.21.15.44:0/4107819650 <== mon.0 v1:172.21.15.44:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f9ccc0146e0 con 0x7f9ce00b45a0 2024-03-25T17:39:06.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.096+0000 7f9cdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:06.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.096+0000 7f9cdd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:06.097 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T17:39:06.097 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:06.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 10 librados: watch_flush enter 2024-03-25T17:39:06.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 10 librados: watch_flush exit 2024-03-25T17:39:06.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 20 client.4164.objecter shutdown clearing up homeless session... 2024-03-25T17:39:06.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 10 client.4164.objecter successfully canceled tick 2024-03-25T17:39:06.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 10 monclient: shutdown 2024-03-25T17:39:06.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:06.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 >> v1:172.21.15.44:6789/0 conn(0x7f9ce00b45a0 legacy=0x7f9ce01d7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:06.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 shutdown_connections 2024-03-25T17:39:06.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 >> 172.21.15.44:0/4107819650 conn(0x7f9ce00b7f40 msgr2=0x7f9ce013f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:06.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.097+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 shutdown_connections 2024-03-25T17:39:06.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.098+0000 7f9ce721e640 1 -- 172.21.15.44:0/4107819650 wait complete. 2024-03-25T17:39:06.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T17:39:06.098+0000 7f9ce721e640 1 librados: shutdown 2024-03-25T17:39:06.114 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T17:39:06.114 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T17:39:06.921 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:07.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.072+0000 7fb140554640 10 monclient: get_monmap_and_config 2024-03-25T17:39:07.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.072+0000 7fb140554640 10 monclient: build_initial_monmap 2024-03-25T17:39:07.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.072+0000 7fb140554640 10 monclient: monmap: 2024-03-25T17:39:07.073 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:07.074 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:07.074 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:39:07.073502+0000 2024-03-25T17:39:07.074 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:39:07.073502+0000 2024-03-25T17:39:07.074 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:07.074 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:07.074 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:07.074 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:07.074 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:07.075 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.073+0000 7fb140554640 1 Processor -- start 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.073+0000 7fb140554640 1 -- start start 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.073+0000 7fb140554640 10 monclient: init 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 10 monclient(hunting): picked mon.noname-b con 0x7fb1380a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 10 monclient(hunting): picked mon.noname-c con 0x7fb13813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 10 monclient(hunting): picked mon.noname-a con 0x7fb1381430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb138142520 con 0x7fb1381430c0 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1380b28a0 con 0x7fb13813f710 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1380b2a30 con 0x7fb1380a98f0 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.074+0000 7fb140554640 10 monclient(hunting): authenticate will time out at 5950.302424s 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.075+0000 7fb13eaf1640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb1381430c0 0x7fb13814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:49578/0 (socket says 172.21.15.44:49578) 2024-03-25T17:39:07.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.075+0000 7fb13eaf1640 1 -- 172.21.15.44:0/3595916914 learned_addr learned my addr 172.21.15.44:0/3595916914 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:07.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.079+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560466297 0 0) 0x7fb138142520 con 0x7fb1381430c0 2024-03-25T17:39:07.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.079+0000 7fb13d2ee640 10 monclient(hunting): _negotiate 2024-03-25T17:39:07.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.079+0000 7fb13d2ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:07.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.079+0000 7fb13d2ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:07.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.079+0000 7fb13d2ee640 10 monclient(hunting): my global_id is 4182 2024-03-25T17:39:07.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.079+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb124003520 con 0x7fb1381430c0 2024-03-25T17:39:07.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.083+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054411841 0 0) 0x7fb124003520 con 0x7fb1381430c0 2024-03-25T17:39:07.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 >> v1:172.21.15.44:6790/0 conn(0x7fb1380a98f0 legacy=0x7fb1380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:39:07.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 >> v1:172.21.15.111:6789/0 conn(0x7fb13813f710 legacy=0x7fb1381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 10 monclient: _finish_hunting 0 2024-03-25T17:39:07.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 1 monclient: found mon.noname-a 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1380b44c0 con 0x7fb1381430c0 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 10 monclient: _finish_auth 0 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 10 monclient: _check_auth_tickets 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb140554640 5 monclient: authenticate success, global_id 4182 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb140554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb130001a10 con 0x7fb1381430c0 2024-03-25T17:39:07.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb13d2ee640 10 monclient: dump: 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:07.088 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:07.089 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:07.089 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:07.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.084+0000 7fb140554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:07.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.087+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb130001ce0 con 0x7fb1381430c0 2024-03-25T17:39:07.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.087+0000 7fb13d2ee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.087+0000 7fb140554640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.087+0000 7fb140554640 4 set_mon_vals no callback set 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb13d2ee640 1 -- 172.21.15.44:0/3595916914 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb130002070 con 0x7fb1381430c0 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb13d2ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb13d2ee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb13d2ee640 10 monclient: dump: 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.094 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb140554640 10 monclient: shutdown 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb140554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb140554640 1 -- 172.21.15.44:0/3595916914 >> v1:172.21.15.44:6789/0 conn(0x7fb1381430c0 legacy=0x7fb13814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb140554640 1 -- 172.21.15.44:0/3595916914 shutdown_connections 2024-03-25T17:39:07.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb140554640 1 -- 172.21.15.44:0/3595916914 >> 172.21.15.44:0/3595916914 conn(0x7fb1380b7ea0 msgr2=0x7fb1380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb140554640 1 -- 172.21.15.44:0/3595916914 shutdown_connections 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.088+0000 7fb140554640 1 -- 172.21.15.44:0/3595916914 wait complete. 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.089+0000 7fb140554640 10 monclient: build_initial_monmap 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.089+0000 7fb140554640 10 monclient: monmap: 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:07.096 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.090+0000 7fb140554640 1 librados: starting msgr at 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.090+0000 7fb140554640 1 librados: starting objecter 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.090+0000 7fb140554640 1 Processor -- start 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.090+0000 7fb140554640 1 -- start start 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.090+0000 7fb140554640 1 librados: setting wanted keys 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.090+0000 7fb140554640 1 librados: calling monclient init 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.090+0000 7fb140554640 10 monclient: init 2024-03-25T17:39:07.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 10 monclient(hunting): picked mon.noname-c con 0x7fb1380a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 10 monclient(hunting): picked mon.noname-b con 0x7fb13813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 10 monclient(hunting): picked mon.noname-a con 0x7fb1381430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1381d7500 con 0x7fb1381430c0 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1380abe50 con 0x7fb13813f710 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1381d8380 con 0x7fb1380a98f0 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb140554640 10 monclient(hunting): authenticate will time out at 5950.319195s 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb13eaf1640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb1381430c0 0x7fb138134140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:49592/0 (socket says 172.21.15.44:49592) 2024-03-25T17:39:07.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.091+0000 7fb13eaf1640 1 -- 172.21.15.44:0/2839565496 learned_addr learned my addr 172.21.15.44:0/2839565496 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:07.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.092+0000 7fb12affd640 10 client.?.objecter ms_handle_connect 0x7fb1381430c0 2024-03-25T17:39:07.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.092+0000 7fb12affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:07.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.092+0000 7fb12affd640 10 client.?.objecter ms_handle_connect 0x7fb13813f710 2024-03-25T17:39:07.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.092+0000 7fb12affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:07.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.095+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45281988 0 0) 0x7fb1381d7500 con 0x7fb1381430c0 2024-03-25T17:39:07.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.095+0000 7fb12affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:07.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.095+0000 7fb12affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:07.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.095+0000 7fb12affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:07.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.095+0000 7fb12affd640 10 monclient(hunting): my global_id is 4185 2024-03-25T17:39:07.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.095+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb11c004eb0 con 0x7fb1381430c0 2024-03-25T17:39:07.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.101+0000 7fb12affd640 10 client.?.objecter ms_handle_connect 0x7fb1380a98f0 2024-03-25T17:39:07.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.101+0000 7fb12affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:07.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.105+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1218494000 0 0) 0x7fb11c004eb0 con 0x7fb1381430c0 2024-03-25T17:39:07.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.105+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1381d7500 con 0x7fb1381430c0 2024-03-25T17:39:07.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.105+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb130001a10 con 0x7fb1381430c0 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.105+0000 7fb12affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.105+0000 7fb12affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.105+0000 7fb12affd640 10 monclient(hunting): dump: 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:07.106 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:07.107 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:07.107 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:07.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.105+0000 7fb12affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.307+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130457295 0 0) 0x7fb1380abe50 con 0x7fb13813f710 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.307+0000 7fb12affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.307+0000 7fb12affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.307+0000 7fb12affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:07.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.307+0000 7fb12affd640 10 monclient(hunting): my global_id is 4171 2024-03-25T17:39:07.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.307+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb11c005680 con 0x7fb13813f710 2024-03-25T17:39:07.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.314+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3744815685 0 0) 0x7fb11c005680 con 0x7fb13813f710 2024-03-25T17:39:07.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.315+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1380abe50 con 0x7fb13813f710 2024-03-25T17:39:07.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.315+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb1340018c0 con 0x7fb13813f710 2024-03-25T17:39:07.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.315+0000 7fb12affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.315+0000 7fb12affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.315+0000 7fb12affd640 10 monclient(hunting): dump: 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:07.316 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:07.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.320+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2851226343 0 0) 0x7fb1380abe50 con 0x7fb13813f710 2024-03-25T17:39:07.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 >> v1:172.21.15.44:6790/0 conn(0x7fb1380a98f0 legacy=0x7fb1381d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 >> v1:172.21.15.44:6789/0 conn(0x7fb1381430c0 legacy=0x7fb138134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 1 monclient: found mon.b 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1381d7750 con 0x7fb13813f710 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 10 monclient: _finish_auth 0 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:07.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb12affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 5 monclient: authenticate success, global_id 4171 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 10 monclient: _renew_subs 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb138138fa0 con 0x7fb13813f710 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 10 client.4171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 10 monclient: _renew_subs 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb138138350 con 0x7fb13813f710 2024-03-25T17:39:07.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.321+0000 7fb140554640 1 librados: init done 2024-03-25T17:39:07.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.322+0000 7fb140554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:07.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.322+0000 7fb140554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:07.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.322+0000 7fb140554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:07.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.322+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1380b3ef0 con 0x7fb13813f710 2024-03-25T17:39:07.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.524+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1340035f0 con 0x7fb13813f710 2024-03-25T17:39:07.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.524+0000 7fb12affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:07.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.524+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb1340039c0 con 0x7fb13813f710 2024-03-25T17:39:07.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.524+0000 7fb12affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.524+0000 7fb12affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.524+0000 7fb12affd640 10 monclient: dump: 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:07.526 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:07.527 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:07.527 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:07.527 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:07.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.524+0000 7fb12bfff640 4 set_mon_vals no callback set 2024-03-25T17:39:07.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.527+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 3) v1 ==== 89499+0+0 (unknown 96833274 0 0) 0x7fb134003be0 con 0x7fb13813f710 2024-03-25T17:39:07.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.533+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fb134053420 con 0x7fb13813f710 2024-03-25T17:39:07.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.533+0000 7fb12affd640 10 client.4171.objecter ms_dispatch 0x7fb138000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:07.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.533+0000 7fb12affd640 3 client.4171.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:07.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.533+0000 7fb12affd640 3 client.4171.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:07.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.533+0000 7fb12affd640 20 client.4171.objecter dump_active .. 0 homeless 2024-03-25T17:39:07.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.533+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb13404ef30 con 0x7fb13813f710 2024-03-25T17:39:07.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.533+0000 7fb12affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:07.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.533+0000 7fb12affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:07.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.655+0000 7fb140554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:07.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.655+0000 7fb140554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:07.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.655+0000 7fb140554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:07.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:07.655+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1380627b0 con 0x7fb13813f710 2024-03-25T17:39:08.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.057+0000 7fb12affd640 1 -- 172.21.15.44:0/2839565496 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fb134050e60 con 0x7fb13813f710 2024-03-25T17:39:08.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.057+0000 7fb12affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:08.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.057+0000 7fb12affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:08.058 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:39:08.058 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:08.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 10 librados: watch_flush enter 2024-03-25T17:39:08.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 10 librados: watch_flush exit 2024-03-25T17:39:08.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 20 client.4171.objecter shutdown clearing up homeless session... 2024-03-25T17:39:08.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 10 client.4171.objecter successfully canceled tick 2024-03-25T17:39:08.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 10 monclient: shutdown 2024-03-25T17:39:08.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:08.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 >> v1:172.21.15.111:6789/0 conn(0x7fb13813f710 legacy=0x7fb138138890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:08.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 shutdown_connections 2024-03-25T17:39:08.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.058+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 >> 172.21.15.44:0/2839565496 conn(0x7fb1380b7ea0 msgr2=0x7fb138074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:08.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.059+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 shutdown_connections 2024-03-25T17:39:08.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.059+0000 7fb140554640 1 -- 172.21.15.44:0/2839565496 wait complete. 2024-03-25T17:39:08.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:08.059+0000 7fb140554640 1 librados: shutdown 2024-03-25T17:39:08.075 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:09.023 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:09.022+0000 7f9c6c8c2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T17:39:09.308 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:39:09.307+0000 7f9c6c8c2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T17:39:12.286 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:06.730 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.287 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:06.737 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.348 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:06.797 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.366 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:06.811 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.434 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:06.864 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.493 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:06.923 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.745 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:07.187 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.807 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:07.250 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.894 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:07.353 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.903 INFO:tasks.ceph.osd.4.smithi111.stderr:2024-03-25T17:39:12.897+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:39:12.919 INFO:tasks.ceph.osd.5.smithi111.stderr:2024-03-25T17:39:12.913+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:39:12.923 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:07.348 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.961 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:07.420 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:12.983 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:07.408 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.053 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:07.502 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.064 INFO:tasks.ceph.osd.6.smithi111.stderr:2024-03-25T17:39:13.058+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:39:13.137 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:07.587 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.318 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:07.752 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.361 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:07.811 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.387 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-03-25T17:39:13.378+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:39:13.394 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:07.829 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.402 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:07.847 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.509 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-03-25T17:39:13.503+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:39:13.549 INFO:tasks.ceph.osd.7.smithi111.stderr:2024-03-25T17:39:13.543+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:39:13.549 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:07.980 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.695 INFO:tasks.ceph.osd.2.smithi044.stderr:2024-03-25T17:39:13.689+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:39:13.737 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:08.186 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.771 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:08.216 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.846 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:08.288 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.929 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:08.360 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:13.963 INFO:tasks.ceph.osd.3.smithi044.stderr:2024-03-25T17:39:13.957+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:39:13.998 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:08.457 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.031 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:08.455 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.098 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:08.547 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.138 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:08.583 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.166 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:08.615 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.213 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:08.655 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.282 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:08.712 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.290 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:08.740 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.375 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:08.834 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.393 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:08.842 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.399 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:08.824 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.428 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:08.862 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.429 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:08.874 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.575 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:09.017 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.583 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:09.014 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.651 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:09.100 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.692 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:09.151 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.706 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:09.155 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.725 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:09.170 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.764 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:09.188 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.797 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:09.231 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.863 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:09.306 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.871 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:09.301 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.882 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:14.951 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:09.401 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.966 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:09.425 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.973+0000 7ff0ecf5c640 10 monclient: get_monmap_and_config 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.973+0000 7ff0ecf5c640 10 monclient: build_initial_monmap 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.973+0000 7ff0ecf5c640 10 monclient: monmap: 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:39:14.974971+0000 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:39:14.974971+0000 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.973+0000 7ff0ecf5c640 1 Processor -- start 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.973+0000 7ff0ecf5c640 1 -- start start 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.973+0000 7ff0ecf5c640 10 monclient: init 2024-03-25T17:39:14.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:14.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:14.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:14.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:14.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e81430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:14.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e8142520 con 0x7ff0e81430c0 2024-03-25T17:39:14.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e80b28a0 con 0x7ff0e80a98f0 2024-03-25T17:39:14.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e80b2a30 con 0x7ff0e813f710 2024-03-25T17:39:14.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:14.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0ecf5c640 10 monclient(hunting): authenticate will time out at 5958.203077s 2024-03-25T17:39:14.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.974+0000 7ff0e6d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff0e81430c0 0x7ff0e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60964/0 (socket says 172.21.15.44:60964) 2024-03-25T17:39:14.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.975+0000 7ff0e6d9d640 1 -- 172.21.15.44:0/3911270346 learned_addr learned my addr 172.21.15.44:0/3911270346 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:14.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341880837 0 0) 0x7ff0e80b28a0 con 0x7ff0e80a98f0 2024-03-25T17:39:14.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 10 monclient(hunting): my global_id is 4198 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d4003520 con 0x7ff0e80a98f0 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664832423 0 0) 0x7ff0e80b2a30 con 0x7ff0e813f710 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 10 monclient(hunting): my global_id is 4199 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.977+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e80b28a0 con 0x7ff0e813f710 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.978+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053645549 0 0) 0x7ff0e8142520 con 0x7ff0e81430c0 2024-03-25T17:39:14.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.978+0000 7ff0e559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.978+0000 7ff0e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.978+0000 7ff0e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.978+0000 7ff0e559a640 10 monclient(hunting): my global_id is 4212 2024-03-25T17:39:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.978+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e80b2a30 con 0x7ff0e81430c0 2024-03-25T17:39:14.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1607469101 0 0) 0x7ff0e80b28a0 con 0x7ff0e813f710 2024-03-25T17:39:14.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 >> v1:172.21.15.111:6789/0 conn(0x7ff0e80a98f0 legacy=0x7ff0e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:14.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 >> v1:172.21.15.44:6789/0 conn(0x7ff0e81430c0 legacy=0x7ff0e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:14.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 10 monclient: _finish_hunting 0 2024-03-25T17:39:14.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 1 monclient: found mon.noname-b 2024-03-25T17:39:14.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80b44c0 con 0x7ff0e813f710 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 10 monclient: _finish_auth 0 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 10 monclient: _check_auth_tickets 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff0d0001670 con 0x7ff0e813f710 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0e559a640 10 monclient: dump: 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.983 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.984 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:14.984 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:14.984 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:14.984 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:14.984 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:14.984 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:14.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0ecf5c640 5 monclient: authenticate success, global_id 4199 2024-03-25T17:39:14.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.980+0000 7ff0ecf5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:14.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.982+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0d00017d0 con 0x7ff0e813f710 2024-03-25T17:39:14.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.982+0000 7ff0e559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:14.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.982+0000 7ff0e559a640 1 -- 172.21.15.44:0/3911270346 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff0d0001c20 con 0x7ff0e813f710 2024-03-25T17:39:14.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.982+0000 7ff0ecf5c640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.982+0000 7ff0e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.982+0000 7ff0ecf5c640 4 set_mon_vals no callback set 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.982+0000 7ff0e559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.982+0000 7ff0e559a640 10 monclient: dump: 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:14.988 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.983+0000 7ff0ecf5c640 10 monclient: shutdown 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.983+0000 7ff0ecf5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.983+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/3911270346 >> v1:172.21.15.44:6790/0 conn(0x7ff0e813f710 legacy=0x7ff0e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.983+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/3911270346 shutdown_connections 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.983+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/3911270346 >> 172.21.15.44:0/3911270346 conn(0x7ff0e80b7ea0 msgr2=0x7ff0e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.983+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/3911270346 shutdown_connections 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.983+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/3911270346 wait complete. 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 10 monclient: build_initial_monmap 2024-03-25T17:39:14.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 10 monclient: monmap: 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 1 librados: starting msgr at 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 1 librados: starting objecter 2024-03-25T17:39:14.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 1 Processor -- start 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 1 -- start start 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 1 librados: setting wanted keys 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 1 librados: calling monclient init 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 10 monclient: init 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.984+0000 7ff0ecf5c640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0ecf5c640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e813f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0ecf5c640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0ecf5c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e80b7c50 con 0x7ff0e813f710 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0ecf5c640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e812dd80 con 0x7ff0e80a98f0 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0ecf5c640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e8139d20 con 0x7ff0e81430c0 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0ecf5c640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0ecf5c640 10 monclient(hunting): authenticate will time out at 5958.213718s 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0e6d9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff0e81430c0 0x7ff0e80b6180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:43072/0 (socket says 172.21.15.44:43072) 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0e6d9d640 1 -- 172.21.15.44:0/1962682437 learned_addr learned my addr 172.21.15.44:0/1962682437 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0deffd640 10 client.?.objecter ms_handle_connect 0x7ff0e81430c0 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0deffd640 10 client.?.objecter ms_handle_connect 0x7ff0e813f710 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0deffd640 10 client.?.objecter ms_handle_connect 0x7ff0e80a98f0 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.985+0000 7ff0deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204436474 0 0) 0x7ff0e8139d20 con 0x7ff0e81430c0 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): my global_id is 4202 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c8004eb0 con 0x7ff0e81430c0 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777322396 0 0) 0x7ff0e812dd80 con 0x7ff0e80a98f0 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:14.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): my global_id is 4201 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e8139d20 con 0x7ff0e80a98f0 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026792136 0 0) 0x7ff0e80b7c50 con 0x7ff0e813f710 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 10 monclient(hunting): my global_id is 4215 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.987+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e812dd80 con 0x7ff0e813f710 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2463344516 0 0) 0x7ff0e8139d20 con 0x7ff0e80a98f0 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0e80b7c50 con 0x7ff0e80a98f0 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff0e0001670 con 0x7ff0e80a98f0 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:14.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): dump: 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438172573 0 0) 0x7ff0c8004eb0 con 0x7ff0e81430c0 2024-03-25T17:39:14.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0e8139d20 con 0x7ff0e81430c0 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff0d8001850 con 0x7ff0e81430c0 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): dump: 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:14.998 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1949310584 0 0) 0x7ff0e812dd80 con 0x7ff0e813f710 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0c8004eb0 con 0x7ff0e813f710 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff0d0001ae0 con 0x7ff0e813f710 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.992+0000 7ff0deffd640 10 monclient(hunting): dump: 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:14.999 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:15.000 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:15.000 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:15.000 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:15.000 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:15.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2840522947 0 0) 0x7ff0e80b7c50 con 0x7ff0e80a98f0 2024-03-25T17:39:15.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 >> v1:172.21.15.44:6790/0 conn(0x7ff0e81430c0 legacy=0x7ff0e80b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:15.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 >> v1:172.21.15.44:6789/0 conn(0x7ff0e813f710 legacy=0x7ff0e80b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:15.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:15.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 1 monclient: found mon.b 2024-03-25T17:39:15.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8138e00 con 0x7ff0e80a98f0 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 5 monclient: authenticate success, global_id 4201 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 10 monclient: _renew_subs 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e8130b20 con 0x7ff0e80a98f0 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 10 monclient: _renew_subs 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:15.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e80b6890 con 0x7ff0e80a98f0 2024-03-25T17:39:15.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.995+0000 7ff0ecf5c640 1 librados: init done 2024-03-25T17:39:15.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.996+0000 7ff0ecf5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:15.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.996+0000 7ff0ecf5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:15.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.996+0000 7ff0ecf5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:15.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:14.996+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e813a430 con 0x7ff0e80a98f0 2024-03-25T17:39:15.021 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:09.470 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.028 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:09.473 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.062 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:09.487 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.115 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:09.549 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.155 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:09.598 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.167 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:09.597 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.210+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0e0003970 con 0x7ff0e80a98f0 2024-03-25T17:39:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.210+0000 7ff0deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.210+0000 7ff0e4d99640 4 set_mon_vals no callback set 2024-03-25T17:39:15.238 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:09.697 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.243 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:09.693 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.251+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff0e0003c80 con 0x7ff0e80a98f0 2024-03-25T17:39:15.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.251+0000 7ff0deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:15.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.251+0000 7ff0deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:15.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.251+0000 7ff0deffd640 10 monclient: dump: 2024-03-25T17:39:15.252 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:15.253 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:15.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.255+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7ff0e0019c50 con 0x7ff0e80a98f0 2024-03-25T17:39:15.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.256+0000 7ff0e5d9b640 10 monclient: get_auth_request con 0x7ff0c806dfc0 auth_method 0 2024-03-25T17:39:15.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.256+0000 7ff0deffd640 10 client.4201.objecter ms_handle_connect 0x7ff0c806dfc0 2024-03-25T17:39:15.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.258+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7ff0e0053920 con 0x7ff0e80a98f0 2024-03-25T17:39:15.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.258+0000 7ff0deffd640 10 client.4201.objecter ms_dispatch 0x7ff0e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:15.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.258+0000 7ff0deffd640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:15.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.258+0000 7ff0deffd640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:15.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.258+0000 7ff0deffd640 20 client.4201.objecter dump_active .. 0 homeless 2024-03-25T17:39:15.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.260+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0e004f260 con 0x7ff0e80a98f0 2024-03-25T17:39:15.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.260+0000 7ff0deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:15.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.260+0000 7ff0deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:15.316 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:09.761 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.337 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:09.786 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.362 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:09.787 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.385 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:09.819 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.444 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:09.887 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.444+0000 7ff0ecf5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:15.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.444+0000 7ff0ecf5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:15.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.444+0000 7ff0ecf5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:15.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.444+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0e80b3ef0 con 0x7ff0e80a98f0 2024-03-25T17:39:15.456 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:09.886 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.459+0000 7ff0deffd640 1 -- 172.21.15.44:0/1962682437 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7ff0e0051360 con 0x7ff0e80a98f0 2024-03-25T17:39:15.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.459+0000 7ff0deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:15.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.459+0000 7ff0deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:15.461 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:39:15.461 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:15.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 10 librados: watch_flush enter 2024-03-25T17:39:15.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 10 librados: watch_flush exit 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 20 client.4201.objecter shutdown clearing up homeless session... 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 10 client.4201.objecter successfully canceled tick 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 >> v1:172.21.15.44:6800/3257043132 conn(0x7ff0c806dfc0 legacy=0x7ff0c8070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 10 monclient: shutdown 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 >> v1:172.21.15.111:6789/0 conn(0x7ff0e80a98f0 legacy=0x7ff0e80b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.461+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 shutdown_connections 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.462+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 >> 172.21.15.44:0/1962682437 conn(0x7ff0e80b7ea0 msgr2=0x7ff0e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.462+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 shutdown_connections 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.462+0000 7ff0ecf5c640 1 -- 172.21.15.44:0/1962682437 wait complete. 2024-03-25T17:39:15.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:15.462+0000 7ff0ecf5c640 1 librados: shutdown 2024-03-25T17:39:15.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:15.510 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:09.969 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.541 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:09.991 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.604 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:10.049 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.634 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:10.083 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.657 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:10.091 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.659 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:10.084 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.741 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:10.184 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.749 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:10.179 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.776 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:10.236 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.789 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:10.232 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.830 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:10.280 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.892 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:10.337 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.926 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:10.361 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.932 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:10.381 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:15.948 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:10.373 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.045 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:10.475 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.046 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:10.505 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.081 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:10.524 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.123 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:10.573 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.190 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:10.625 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.248 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:10.673 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.309 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:10.769 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.408 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:10.853 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.419 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:10.869 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.457 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:10.892 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.463 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:10.908 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.466 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:10.915 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.532 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:10.981 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.546 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:10.971 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.590 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:11.020 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.604 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:11.047 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.655 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:11.085 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.660 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:11.102 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.734 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:11.169 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.816 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:11.276 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.872 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:11.331 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:16.972 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:11.422 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:17.048 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:11.498 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:17.092 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:11.517 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:17.162 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:11.587 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:17.248 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:11.682 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:17.308 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:11.742 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:21.884 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:16.328 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:22.216 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:16.658 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:22.279 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:22.349 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:16.780 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:22.376 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:16.825 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:22.404 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:16.863 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:22.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.415+0000 7fe39cfbc640 10 monclient: get_monmap_and_config 2024-03-25T17:39:22.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.415+0000 7fe39cfbc640 10 monclient: build_initial_monmap 2024-03-25T17:39:22.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.415+0000 7fe39cfbc640 10 monclient: monmap: 2024-03-25T17:39:22.415 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:39:22.416166+0000 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:39:22.416166+0000 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:22.416 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.415+0000 7fe39cfbc640 1 Processor -- start 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.415+0000 7fe39cfbc640 1 -- start start 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.415+0000 7fe39cfbc640 10 monclient: init 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 10 monclient(hunting): picked mon.noname-a con 0x7fe3980a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 10 monclient(hunting): picked mon.noname-c con 0x7fe39813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 10 monclient(hunting): picked mon.noname-b con 0x7fe3981430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe398142520 con 0x7fe3980a98f0 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3980b28a0 con 0x7fe39813f710 2024-03-25T17:39:22.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3980b2a30 con 0x7fe3981430c0 2024-03-25T17:39:22.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:22.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.416+0000 7fe39cfbc640 10 monclient(hunting): authenticate will time out at 5965.644566s 2024-03-25T17:39:22.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.417+0000 7fe396d9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fe3981430c0 0x7fe39814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:50910/0 (socket says 172.21.15.44:50910) 2024-03-25T17:39:22.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.417+0000 7fe396d9d640 1 -- 172.21.15.44:0/2243200584 learned_addr learned my addr 172.21.15.44:0/2243200584 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:22.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.420+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573007623 0 0) 0x7fe3980b28a0 con 0x7fe39813f710 2024-03-25T17:39:22.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.420+0000 7fe39559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.420+0000 7fe39559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.420+0000 7fe39559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.420+0000 7fe39559a640 10 monclient(hunting): my global_id is 4204 2024-03-25T17:39:22.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.420+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe380003520 con 0x7fe39813f710 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775235153 0 0) 0x7fe3980b2a30 con 0x7fe3981430c0 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 10 monclient(hunting): my global_id is 4205 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3980b28a0 con 0x7fe3981430c0 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245342014 0 0) 0x7fe398142520 con 0x7fe3980a98f0 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 10 monclient(hunting): my global_id is 4218 2024-03-25T17:39:22.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.421+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3980b2a30 con 0x7fe3980a98f0 2024-03-25T17:39:22.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309689468 0 0) 0x7fe380003520 con 0x7fe39813f710 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 >> v1:172.21.15.44:6790/0 conn(0x7fe3981430c0 legacy=0x7fe39814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 >> v1:172.21.15.44:6789/0 conn(0x7fe3980a98f0 legacy=0x7fe3980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 10 monclient: _finish_hunting 0 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 1 monclient: found mon.noname-c 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3980b44c0 con 0x7fe39813f710 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 10 monclient: _finish_auth 0 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 10 monclient: _check_auth_tickets 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39cfbc640 5 monclient: authenticate success, global_id 4204 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39cfbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe38c001650 con 0x7fe39813f710 2024-03-25T17:39:22.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39559a640 10 monclient: dump: 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:22.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.425+0000 7fe39cfbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.427+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe38c0018a0 con 0x7fe39813f710 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.427+0000 7fe39559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.427+0000 7fe39559a640 1 -- 172.21.15.44:0/2243200584 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe38c001c30 con 0x7fe39813f710 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.427+0000 7fe39559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.427+0000 7fe39559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.427+0000 7fe39559a640 10 monclient: dump: 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.428 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.429 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:22.429 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:22.429 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:22.429 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:22.429 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:22.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.427+0000 7fe39cfbc640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:22.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.427+0000 7fe39cfbc640 4 set_mon_vals no callback set 2024-03-25T17:39:22.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.428+0000 7fe39cfbc640 10 monclient: shutdown 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.428+0000 7fe39cfbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.428+0000 7fe39cfbc640 1 -- 172.21.15.44:0/2243200584 >> v1:172.21.15.111:6789/0 conn(0x7fe39813f710 legacy=0x7fe3981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.428+0000 7fe39cfbc640 1 -- 172.21.15.44:0/2243200584 shutdown_connections 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.428+0000 7fe39cfbc640 1 -- 172.21.15.44:0/2243200584 >> 172.21.15.44:0/2243200584 conn(0x7fe3980b7ea0 msgr2=0x7fe3980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.428+0000 7fe39cfbc640 1 -- 172.21.15.44:0/2243200584 shutdown_connections 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.428+0000 7fe39cfbc640 1 -- 172.21.15.44:0/2243200584 wait complete. 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 10 monclient: build_initial_monmap 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 10 monclient: monmap: 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:22.431 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 1 librados: starting msgr at 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 1 librados: starting objecter 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 1 Processor -- start 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 1 -- start start 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 1 librados: setting wanted keys 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 1 librados: calling monclient init 2024-03-25T17:39:22.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.430+0000 7fe39cfbc640 10 monclient: init 2024-03-25T17:39:22.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:22.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:22.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 10 monclient(hunting): picked mon.noname-c con 0x7fe3980a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:22.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 10 monclient(hunting): picked mon.noname-b con 0x7fe39813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:22.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 10 monclient(hunting): picked mon.noname-a con 0x7fe3981430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:22.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe398131c30 con 0x7fe3981430c0 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe398137d20 con 0x7fe39813f710 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3981319b0 con 0x7fe3980a98f0 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.431+0000 7fe39cfbc640 10 monclient(hunting): authenticate will time out at 5965.659577s 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.432+0000 7fe395d9b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe39813f710 0x7fe398130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52718/0 (socket says 172.21.15.44:52718) 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.432+0000 7fe395d9b640 1 -- 172.21.15.44:0/999338991 learned_addr learned my addr 172.21.15.44:0/999338991 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.432+0000 7fe386ffd640 10 client.?.objecter ms_handle_connect 0x7fe39813f710 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.432+0000 7fe386ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.432+0000 7fe386ffd640 10 client.?.objecter ms_handle_connect 0x7fe3980a98f0 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.432+0000 7fe386ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.432+0000 7fe386ffd640 10 client.?.objecter ms_handle_connect 0x7fe3981430c0 2024-03-25T17:39:22.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.432+0000 7fe386ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:22.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.434+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911698889 0 0) 0x7fe398131c30 con 0x7fe3981430c0 2024-03-25T17:39:22.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 10 monclient(hunting): my global_id is 4221 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe378004eb0 con 0x7fe3981430c0 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634862591 0 0) 0x7fe398137d20 con 0x7fe39813f710 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 10 monclient(hunting): my global_id is 4207 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.435+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe398131c30 con 0x7fe39813f710 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.436+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271040104 0 0) 0x7fe3981319b0 con 0x7fe3980a98f0 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.436+0000 7fe386ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.436+0000 7fe386ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.436+0000 7fe386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.436+0000 7fe386ffd640 10 monclient(hunting): my global_id is 4208 2024-03-25T17:39:22.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.436+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe398137d20 con 0x7fe3980a98f0 2024-03-25T17:39:22.439 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:16.864 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:22.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2785241317 0 0) 0x7fe378004eb0 con 0x7fe3981430c0 2024-03-25T17:39:22.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3981319b0 con 0x7fe3981430c0 2024-03-25T17:39:22.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe390001ab0 con 0x7fe3981430c0 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:22.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2950210654 0 0) 0x7fe398131c30 con 0x7fe39813f710 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe378004eb0 con 0x7fe39813f710 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 705254423 0 0) 0x7fe398137d20 con 0x7fe3980a98f0 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe398131c30 con 0x7fe3980a98f0 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe388001830 con 0x7fe3980a98f0 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.442+0000 7fe386ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:22.445 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.443+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe38c001cd0 con 0x7fe39813f710 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.443+0000 7fe386ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.443+0000 7fe386ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.443+0000 7fe386ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:22.446 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2401236912 0 0) 0x7fe398131c30 con 0x7fe3980a98f0 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 >> v1:172.21.15.111:6789/0 conn(0x7fe39813f710 legacy=0x7fe398130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 >> v1:172.21.15.44:6789/0 conn(0x7fe3981430c0 legacy=0x7fe3981312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 1 monclient: found mon.c 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe398139470 con 0x7fe3980a98f0 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe386ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:22.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 5 monclient: authenticate success, global_id 4208 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 10 monclient: _renew_subs 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe398139750 con 0x7fe3980a98f0 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 10 monclient: _renew_subs 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3980b7a80 con 0x7fe3980a98f0 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.447+0000 7fe39cfbc640 1 librados: init done 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.449+0000 7fe39cfbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:22.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.449+0000 7fe39cfbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:22.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.449+0000 7fe39cfbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:22.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.449+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe398137ad0 con 0x7fe3980a98f0 2024-03-25T17:39:22.815 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:17.249 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:22.867 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:17.317 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.896+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe388001d40 con 0x7fe3980a98f0 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.896+0000 7fe386ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.896+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe3880037b0 con 0x7fe3980a98f0 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.896+0000 7fe386ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.896+0000 7fe386ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.896+0000 7fe386ffd640 10 monclient: dump: 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:22.897 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.898 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:22.898 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:22.898 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:22.898 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:22.898 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:22.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.896+0000 7fe394d99640 4 set_mon_vals no callback set 2024-03-25T17:39:22.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.897+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fe388019840 con 0x7fe3980a98f0 2024-03-25T17:39:22.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.898+0000 7fe395d9b640 10 monclient: get_auth_request con 0x7fe37806e090 auth_method 0 2024-03-25T17:39:22.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.898+0000 7fe386ffd640 10 client.4208.objecter ms_handle_connect 0x7fe37806e090 2024-03-25T17:39:22.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.903+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fe3880534e0 con 0x7fe3980a98f0 2024-03-25T17:39:22.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.903+0000 7fe386ffd640 10 client.4208.objecter ms_dispatch 0x7fe398000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:22.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.903+0000 7fe386ffd640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:22.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.903+0000 7fe386ffd640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:22.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.903+0000 7fe386ffd640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T17:39:22.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.903+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe38804ee20 con 0x7fe3980a98f0 2024-03-25T17:39:22.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.903+0000 7fe386ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:22.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:22.903+0000 7fe386ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:23.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.089+0000 7fe39cfbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:23.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.089+0000 7fe39cfbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:23.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.089+0000 7fe39cfbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:23.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.089+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3980627b0 con 0x7fe3980a98f0 2024-03-25T17:39:23.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.169+0000 7fe386ffd640 1 -- 172.21.15.44:0/999338991 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fe388050f20 con 0x7fe3980a98f0 2024-03-25T17:39:23.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.169+0000 7fe386ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:23.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.169+0000 7fe386ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:23.170 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:39:23.170 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.171+0000 7fe39cfbc640 10 librados: watch_flush enter 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 10 librados: watch_flush exit 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 10 client.4208.objecter successfully canceled tick 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 >> v1:172.21.15.44:6800/3257043132 conn(0x7fe37806e090 legacy=0x7fe378070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 10 monclient: shutdown 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 >> v1:172.21.15.44:6790/0 conn(0x7fe3980a98f0 legacy=0x7fe3981db2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 shutdown_connections 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 >> 172.21.15.44:0/999338991 conn(0x7fe3980b7ea0 msgr2=0x7fe398074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 shutdown_connections 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 1 -- 172.21.15.44:0/999338991 wait complete. 2024-03-25T17:39:23.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:23.172+0000 7fe39cfbc640 1 librados: shutdown 2024-03-25T17:39:23.184 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:27.391 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:21.835 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:27.394 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:21.839 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:27.570 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:22.000 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:27.575 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:22.005 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:27.687 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:22.137 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:27.693 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:22.143 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:27.785 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:22.209 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:27.790 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:22.215 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:28.427 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:22.887 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:28.431 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:22.891 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:28.774 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:23.209 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:28.778 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:23.213 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:28.839 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:23.282 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:28.843 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:23.286 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:29.097 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:23.547 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:29.103 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:23.553 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:29.991 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.120+0000 7f32e5a96640 10 monclient: get_monmap_and_config 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.120+0000 7f32e5a96640 10 monclient: build_initial_monmap 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.121+0000 7f32e5a96640 10 monclient: monmap: 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:39:30.121926+0000 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:39:30.121926+0000 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:30.121 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:30.122 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:30.122 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.121+0000 7f32e5a96640 1 Processor -- start 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.121+0000 7f32e5a96640 1 -- start start 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.121+0000 7f32e5a96640 10 monclient: init 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 10 monclient(hunting): picked mon.noname-c con 0x7f32e00b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 10 monclient(hunting): picked mon.noname-b con 0x7f32e00b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 10 monclient(hunting): picked mon.noname-a con 0x7f32e014be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:30.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00af940 con 0x7f32e014be90 2024-03-25T17:39:30.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00b0a70 con 0x7f32e00b4d60 2024-03-25T17:39:30.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00b0d50 con 0x7f32e00b1730 2024-03-25T17:39:30.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:30.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.122+0000 7f32e5a96640 10 monclient(hunting): authenticate will time out at 5973.350368s 2024-03-25T17:39:30.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.123+0000 7f32deffd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f32e00b1730 0x7f32e00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:48534/0 (socket says 172.21.15.44:48534) 2024-03-25T17:39:30.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.123+0000 7f32deffd640 1 -- 172.21.15.44:0/3456095955 learned_addr learned my addr 172.21.15.44:0/3456095955 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:30.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063615428 0 0) 0x7f32e00af940 con 0x7f32e014be90 2024-03-25T17:39:30.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 10 monclient(hunting): my global_id is 4224 2024-03-25T17:39:30.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c8003520 con 0x7f32e014be90 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154623637 0 0) 0x7f32e00b0a70 con 0x7f32e00b4d60 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 10 monclient(hunting): my global_id is 4210 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.126+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e00af940 con 0x7f32e00b4d60 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.127+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436988542 0 0) 0x7f32e00b0d50 con 0x7f32e00b1730 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.127+0000 7f32de7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.127+0000 7f32de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.127+0000 7f32de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.127+0000 7f32de7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T17:39:30.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.127+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c80039d0 con 0x7f32e00b1730 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.130+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3834347380 0 0) 0x7f32c8003520 con 0x7f32e014be90 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.130+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 >> v1:172.21.15.44:6790/0 conn(0x7f32e00b1730 legacy=0x7f32e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.130+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 >> v1:172.21.15.111:6789/0 conn(0x7f32e00b4d60 legacy=0x7f32e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 1 monclient: found mon.noname-a 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00b1030 con 0x7f32e014be90 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 10 monclient: _finish_auth 0 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:39:30.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f32d8001a10 con 0x7f32e014be90 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32de7fc640 10 monclient: dump: 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32e5a96640 5 monclient: authenticate success, global_id 4224 2024-03-25T17:39:30.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.131+0000 7f32e5a96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:30.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.133+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32d8001b70 con 0x7f32e014be90 2024-03-25T17:39:30.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.133+0000 7f32de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:30.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.133+0000 7f32de7fc640 1 -- 172.21.15.44:0/3456095955 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f32d8001f80 con 0x7f32e014be90 2024-03-25T17:39:30.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.133+0000 7f32de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.133+0000 7f32de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:30.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.133+0000 7f32de7fc640 10 monclient: dump: 2024-03-25T17:39:30.133 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:30.133 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.133+0000 7f32e5a96640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:30.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.133+0000 7f32e5a96640 4 set_mon_vals no callback set 2024-03-25T17:39:30.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.134+0000 7f32e5a96640 10 monclient: shutdown 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.134+0000 7f32e5a96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.134+0000 7f32e5a96640 1 -- 172.21.15.44:0/3456095955 >> v1:172.21.15.44:6789/0 conn(0x7f32e014be90 legacy=0x7f32e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.134+0000 7f32e5a96640 1 -- 172.21.15.44:0/3456095955 shutdown_connections 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.134+0000 7f32e5a96640 1 -- 172.21.15.44:0/3456095955 >> 172.21.15.44:0/3456095955 conn(0x7f32e00617a0 msgr2=0x7f32e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.134+0000 7f32e5a96640 1 -- 172.21.15.44:0/3456095955 shutdown_connections 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.134+0000 7f32e5a96640 1 -- 172.21.15.44:0/3456095955 wait complete. 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 10 monclient: build_initial_monmap 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 10 monclient: monmap: 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:30.136 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 1 librados: starting msgr at 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 1 librados: starting objecter 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 1 Processor -- start 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 1 -- start start 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 1 librados: setting wanted keys 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 1 librados: calling monclient init 2024-03-25T17:39:30.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.135+0000 7f32e5a96640 10 monclient: init 2024-03-25T17:39:30.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:30.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:30.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 10 monclient(hunting): picked mon.noname-b con 0x7f32e00b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:30.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 10 monclient(hunting): picked mon.noname-a con 0x7f32e00b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:30.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 10 monclient(hunting): picked mon.noname-c con 0x7f32e014be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:30.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e01dc4b0 con 0x7f32e00b4d60 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e0134f90 con 0x7f32e00b1730 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e01dd050 con 0x7f32e014be90 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.136+0000 7f32e5a96640 10 monclient(hunting): authenticate will time out at 5973.364263s 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.137+0000 7f32dffff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f32e014be90 0x7f32e0134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:48536/0 (socket says 172.21.15.44:48536) 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.137+0000 7f32dffff640 1 -- 172.21.15.44:0/960681215 learned_addr learned my addr 172.21.15.44:0/960681215 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.137+0000 7f32c7fff640 10 client.?.objecter ms_handle_connect 0x7f32e014be90 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.137+0000 7f32c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.137+0000 7f32c7fff640 10 client.?.objecter ms_handle_connect 0x7f32e00b4d60 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.137+0000 7f32c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.137+0000 7f32c7fff640 10 client.?.objecter ms_handle_connect 0x7f32e00b1730 2024-03-25T17:39:30.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.137+0000 7f32c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:30.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851238098 0 0) 0x7f32e0134f90 con 0x7f32e00b1730 2024-03-25T17:39:30.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): my global_id is 4213 2024-03-25T17:39:30.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32cc004eb0 con 0x7f32e00b1730 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261059883 0 0) 0x7f32e01dd050 con 0x7f32e014be90 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): my global_id is 4214 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e0134f90 con 0x7f32e014be90 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100815156 0 0) 0x7f32e01dc4b0 con 0x7f32e00b4d60 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 10 monclient(hunting): my global_id is 4227 2024-03-25T17:39:30.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.140+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e01dd050 con 0x7f32e00b4d60 2024-03-25T17:39:30.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 577169375 0 0) 0x7f32e0134f90 con 0x7f32e014be90 2024-03-25T17:39:30.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32e01dc4b0 con 0x7f32e014be90 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 941339215 0 0) 0x7f32e01dd050 con 0x7f32e00b4d60 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32e0134f90 con 0x7f32e00b4d60 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f32d8001ee0 con 0x7f32e014be90 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 10 monclient(hunting): dump: 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:30.146 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f32d4001870 con 0x7f32e00b4d60 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.145+0000 7f32c7fff640 10 monclient(hunting): dump: 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:30.147 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.148 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.148 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.148 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:30.148 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:30.148 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:30.148 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:30.148 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.148+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359933420 0 0) 0x7f32cc004eb0 con 0x7f32e00b1730 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.148+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32e01dd050 con 0x7f32e00b1730 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.148+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f32d0001950 con 0x7f32e00b1730 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.148+0000 7f32c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.148+0000 7f32c7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.148+0000 7f32c7fff640 10 monclient(hunting): dump: 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.149 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1053894350 0 0) 0x7f32e0134f90 con 0x7f32e00b4d60 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 >> v1:172.21.15.44:6790/0 conn(0x7f32e014be90 legacy=0x7f32e0134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 >> v1:172.21.15.111:6789/0 conn(0x7f32e00b1730 legacy=0x7f32e01dc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 1 monclient: found mon.a 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e013b210 con 0x7f32e00b4d60 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 10 monclient: _finish_auth 0 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 5 monclient: authenticate success, global_id 4227 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 10 monclient: _renew_subs 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e013b4c0 con 0x7f32e00b4d60 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 10 monclient: _renew_subs 2024-03-25T17:39:30.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:30.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e01407d0 con 0x7f32e00b4d60 2024-03-25T17:39:30.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.149+0000 7f32e5a96640 1 librados: init done 2024-03-25T17:39:30.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.151+0000 7f32e5a96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:30.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.151+0000 7f32e5a96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:30.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.151+0000 7f32e5a96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:30.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.151+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e01401f0 con 0x7f32e00b4d60 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.168+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32d40047e0 con 0x7f32e00b4d60 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.168+0000 7f32c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.168+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f32d4001d80 con 0x7f32e00b4d60 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.168+0000 7f32c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.168+0000 7f32c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.168+0000 7f32c7fff640 10 monclient: dump: 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.169 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:30.170 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:30.170 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:30.170 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:30.170 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:30.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.168+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f32d401a7c0 con 0x7f32e00b4d60 2024-03-25T17:39:30.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.168+0000 7f32ddffb640 4 set_mon_vals no callback set 2024-03-25T17:39:30.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.169+0000 7f32df7fe640 10 monclient: get_auth_request con 0x7f32cc06e2c0 auth_method 0 2024-03-25T17:39:30.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.169+0000 7f32c7fff640 10 client.4227.objecter ms_handle_connect 0x7f32cc06e2c0 2024-03-25T17:39:30.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.170+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f32d4054610 con 0x7f32e00b4d60 2024-03-25T17:39:30.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.170+0000 7f32c7fff640 10 client.4227.objecter ms_dispatch 0x7f32e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:30.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.170+0000 7f32c7fff640 3 client.4227.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:30.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.170+0000 7f32c7fff640 3 client.4227.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:30.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.170+0000 7f32c7fff640 20 client.4227.objecter dump_active .. 0 homeless 2024-03-25T17:39:30.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.415+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32d404ff50 con 0x7f32e00b4d60 2024-03-25T17:39:30.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.415+0000 7f32c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:30.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.415+0000 7f32c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:30.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.694+0000 7f32e5a96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:30.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.694+0000 7f32e5a96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:30.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.694+0000 7f32e5a96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:30.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.694+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32e01dc660 con 0x7f32e00b4d60 2024-03-25T17:39:30.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.720+0000 7f32c7fff640 1 -- 172.21.15.44:0/960681215 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f32e01dc660 con 0x7f32e00b4d60 2024-03-25T17:39:30.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.720+0000 7f32c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:30.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.720+0000 7f32c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:30.720 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:39:30.720 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:30.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.721+0000 7f32e5a96640 10 librados: watch_flush enter 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 10 librados: watch_flush exit 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 20 client.4227.objecter shutdown clearing up homeless session... 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 10 client.4227.objecter successfully canceled tick 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 >> v1:172.21.15.44:6800/3257043132 conn(0x7f32cc06e2c0 legacy=0x7f32cc070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 10 monclient: shutdown 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 >> v1:172.21.15.44:6789/0 conn(0x7f32e00b4d60 legacy=0x7f32e0134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 shutdown_connections 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 >> 172.21.15.44:0/960681215 conn(0x7f32e00617a0 msgr2=0x7f32e013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 shutdown_connections 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 1 -- 172.21.15.44:0/960681215 wait complete. 2024-03-25T17:39:30.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:30.722+0000 7f32e5a96640 1 librados: shutdown 2024-03-25T17:39:30.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:31.472 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:25.917 35159== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:31.762 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:26.212 35158== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:31.919 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:26.350 35160== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:32.027 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:26.452 35161== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:33.185 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:27.627 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:33.292 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:27.752 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:33.474 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:27.924 35504== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:33.764 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:28.198 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T17:39:37.543 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:37.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.673+0000 7f7a60fb5640 10 monclient: get_monmap_and_config 2024-03-25T17:39:37.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.673+0000 7f7a60fb5640 10 monclient: build_initial_monmap 2024-03-25T17:39:37.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.673+0000 7f7a60fb5640 10 monclient: monmap: 2024-03-25T17:39:37.674 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:37.674 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:37.674 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:39:37.674821+0000 2024-03-25T17:39:37.674 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:39:37.674821+0000 2024-03-25T17:39:37.675 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:37.675 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:37.675 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:37.675 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:37.675 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:37.675 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:37.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.674+0000 7f7a60fb5640 1 Processor -- start 2024-03-25T17:39:37.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.674+0000 7f7a60fb5640 1 -- start start 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.674+0000 7f7a60fb5640 10 monclient: init 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f7a5c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f7a5c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f7a5c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c142520 con 0x7f7a5c1430c0 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c0b28a0 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c0b2a30 con 0x7f7a5c13f710 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.675+0000 7f7a60fb5640 10 monclient(hunting): authenticate will time out at 5980.903179s 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.676+0000 7f7a5a59c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7a5c0a98f0 0x7f7a5c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:34730/0 (socket says 172.21.15.44:34730) 2024-03-25T17:39:37.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.676+0000 7f7a5a59c640 1 -- 172.21.15.44:0/3293347939 learned_addr learned my addr 172.21.15.44:0/3293347939 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:37.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.678+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628344991 0 0) 0x7f7a5c0b2a30 con 0x7f7a5c13f710 2024-03-25T17:39:37.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.678+0000 7f7a5959a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.678+0000 7f7a5959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.678+0000 7f7a5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.678+0000 7f7a5959a640 10 monclient(hunting): my global_id is 4217 2024-03-25T17:39:37.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.678+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a44003520 con 0x7f7a5c13f710 2024-03-25T17:39:37.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048578631 0 0) 0x7f7a5c142520 con 0x7f7a5c1430c0 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 10 monclient(hunting): my global_id is 4230 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c0b2a30 con 0x7f7a5c1430c0 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081714118 0 0) 0x7f7a5c0b28a0 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 10 monclient(hunting): my global_id is 4216 2024-03-25T17:39:37.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.679+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c142520 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3545753830 0 0) 0x7f7a44003520 con 0x7f7a5c13f710 2024-03-25T17:39:37.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 >> v1:172.21.15.111:6789/0 conn(0x7f7a5c0a98f0 legacy=0x7f7a5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 >> v1:172.21.15.44:6789/0 conn(0x7f7a5c1430c0 legacy=0x7f7a5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 10 monclient: _finish_hunting 0 2024-03-25T17:39:37.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 1 monclient: found mon.noname-b 2024-03-25T17:39:37.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:37.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:39:37.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c0b44c0 con 0x7f7a5c13f710 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 10 monclient: _finish_auth 0 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 10 monclient: _check_auth_tickets 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a50001690 con 0x7f7a5c13f710 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a60fb5640 5 monclient: authenticate success, global_id 4217 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a5959a640 10 monclient: dump: 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:37.684 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:37.685 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:37.685 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:37.685 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:37.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.682+0000 7f7a60fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:37.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.684+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a500017f0 con 0x7f7a5c13f710 2024-03-25T17:39:37.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.684+0000 7f7a5959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:37.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.684+0000 7f7a5959a640 1 -- 172.21.15.44:0/3293347939 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a50001c00 con 0x7f7a5c13f710 2024-03-25T17:39:37.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.684+0000 7f7a5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.684+0000 7f7a5959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:37.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.684+0000 7f7a5959a640 10 monclient: dump: 2024-03-25T17:39:37.687 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:37.687 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.684+0000 7f7a60fb5640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.684+0000 7f7a60fb5640 4 set_mon_vals no callback set 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.685+0000 7f7a60fb5640 10 monclient: shutdown 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.685+0000 7f7a60fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.685+0000 7f7a60fb5640 1 -- 172.21.15.44:0/3293347939 >> v1:172.21.15.44:6790/0 conn(0x7f7a5c13f710 legacy=0x7f7a5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.685+0000 7f7a60fb5640 1 -- 172.21.15.44:0/3293347939 shutdown_connections 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.685+0000 7f7a60fb5640 1 -- 172.21.15.44:0/3293347939 >> 172.21.15.44:0/3293347939 conn(0x7f7a5c0b7ea0 msgr2=0x7f7a5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:37.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.685+0000 7f7a60fb5640 1 -- 172.21.15.44:0/3293347939 shutdown_connections 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.685+0000 7f7a60fb5640 1 -- 172.21.15.44:0/3293347939 wait complete. 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.686+0000 7f7a60fb5640 10 monclient: build_initial_monmap 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.686+0000 7f7a60fb5640 10 monclient: monmap: 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 librados: starting msgr at 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 librados: starting objecter 2024-03-25T17:39:37.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 Processor -- start 2024-03-25T17:39:37.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 -- start start 2024-03-25T17:39:37.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 librados: setting wanted keys 2024-03-25T17:39:37.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 librados: calling monclient init 2024-03-25T17:39:37.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 10 monclient: init 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f7a5c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f7a5c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f7a5c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c136ef0 con 0x7f7a5c13f710 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c136b30 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c136cc0 con 0x7f7a5c1430c0 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.687+0000 7f7a60fb5640 10 monclient(hunting): authenticate will time out at 5980.915612s 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.688+0000 7f7a5a59c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7a5c0a98f0 0x7f7a5c1d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:34740/0 (socket says 172.21.15.44:34740) 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.688+0000 7f7a5a59c640 1 -- 172.21.15.44:0/2752274473 learned_addr learned my addr 172.21.15.44:0/2752274473 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.688+0000 7f7a4affd640 10 client.?.objecter ms_handle_connect 0x7f7a5c0a98f0 2024-03-25T17:39:37.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.688+0000 7f7a4affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.688+0000 7f7a4affd640 10 client.?.objecter ms_handle_connect 0x7f7a5c1430c0 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.688+0000 7f7a4affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.689+0000 7f7a4affd640 10 client.?.objecter ms_handle_connect 0x7f7a5c13f710 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.689+0000 7f7a4affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649650948 0 0) 0x7f7a5c136cc0 con 0x7f7a5c1430c0 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 10 monclient(hunting): my global_id is 4220 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a3c004eb0 con 0x7f7a5c1430c0 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175477003 0 0) 0x7f7a5c136b30 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 10 monclient(hunting): my global_id is 4219 2024-03-25T17:39:37.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.691+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c136cc0 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.692+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771082758 0 0) 0x7f7a5c136ef0 con 0x7f7a5c13f710 2024-03-25T17:39:37.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.692+0000 7f7a4affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.692+0000 7f7a4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.692+0000 7f7a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.692+0000 7f7a4affd640 10 monclient(hunting): my global_id is 4233 2024-03-25T17:39:37.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.692+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c136b30 con 0x7f7a5c13f710 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.698+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2714060018 0 0) 0x7f7a5c136cc0 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.698+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a5c136ef0 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.698+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028901215 0 0) 0x7f7a3c004eb0 con 0x7f7a5c1430c0 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.698+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a5c136cc0 con 0x7f7a5c1430c0 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.698+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a54001b50 con 0x7f7a5c1430c0 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.698+0000 7f7a4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.698+0000 7f7a4affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.698+0000 7f7a4affd640 10 monclient(hunting): dump: 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.700 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a4c001a50 con 0x7f7a5c0a98f0 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 10 monclient(hunting): dump: 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.701 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 854540827 0 0) 0x7f7a5c136b30 con 0x7f7a5c13f710 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a3c004eb0 con 0x7f7a5c13f710 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a50001b00 con 0x7f7a5c13f710 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.699+0000 7f7a4affd640 10 monclient(hunting): dump: 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:37.702 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:37.703 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:37.703 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:37.703 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2562051778 0 0) 0x7f7a5c136cc0 con 0x7f7a5c1430c0 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 >> v1:172.21.15.111:6789/0 conn(0x7f7a5c0a98f0 legacy=0x7f7a5c1d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 >> v1:172.21.15.44:6789/0 conn(0x7f7a5c13f710 legacy=0x7f7a5c0b6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 1 monclient: found mon.c 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c139690 con 0x7f7a5c1430c0 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 10 monclient: _finish_auth 0 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.703+0000 7f7a60fb5640 5 monclient: authenticate success, global_id 4220 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.704+0000 7f7a60fb5640 10 monclient: _renew_subs 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.704+0000 7f7a60fb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:37.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.704+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5c139940 con 0x7f7a5c1430c0 2024-03-25T17:39:37.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.704+0000 7f7a60fb5640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:37.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.704+0000 7f7a60fb5640 10 monclient: _renew_subs 2024-03-25T17:39:37.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.704+0000 7f7a60fb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:37.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.704+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5c133880 con 0x7f7a5c1430c0 2024-03-25T17:39:37.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.704+0000 7f7a60fb5640 1 librados: init done 2024-03-25T17:39:37.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.705+0000 7f7a60fb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:37.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.706+0000 7f7a60fb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:37.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.706+0000 7f7a60fb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:37.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.706+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5c074cc0 con 0x7f7a5c1430c0 2024-03-25T17:39:37.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.707+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a54002040 con 0x7f7a5c1430c0 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.707+0000 7f7a4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.707+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a54003b10 con 0x7f7a5c1430c0 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.707+0000 7f7a58d99640 4 set_mon_vals no callback set 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.707+0000 7f7a4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.707+0000 7f7a4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.707+0000 7f7a4affd640 10 monclient: dump: 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:37.708 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:37.709 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:37.709 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:37.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.803+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f7a54019ae0 con 0x7f7a5c1430c0 2024-03-25T17:39:37.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.803+0000 7f7a5a59c640 10 monclient: get_auth_request con 0x7f7a3c06e0c0 auth_method 0 2024-03-25T17:39:37.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.804+0000 7f7a4affd640 10 client.4220.objecter ms_handle_connect 0x7f7a3c06e0c0 2024-03-25T17:39:37.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.980+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f7a54053800 con 0x7f7a5c1430c0 2024-03-25T17:39:37.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.980+0000 7f7a4affd640 10 client.4220.objecter ms_dispatch 0x7f7a5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:37.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.980+0000 7f7a4affd640 3 client.4220.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:37.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.980+0000 7f7a4affd640 3 client.4220.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:37.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.980+0000 7f7a4affd640 20 client.4220.objecter dump_active .. 0 homeless 2024-03-25T17:39:37.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.980+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a5404f140 con 0x7f7a5c1430c0 2024-03-25T17:39:37.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.980+0000 7f7a4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:37.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:37.980+0000 7f7a4affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:38.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.355+0000 7f7a60fb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:38.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.355+0000 7f7a60fb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:38.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.355+0000 7f7a60fb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:39:38.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.355+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a5c0b7610 con 0x7f7a5c1430c0 2024-03-25T17:39:38.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.370+0000 7f7a4affd640 1 -- 172.21.15.44:0/2752274473 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f7a54051240 con 0x7f7a5c1430c0 2024-03-25T17:39:38.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.370+0000 7f7a4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:38.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.370+0000 7f7a4affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:38.370 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:39:38.371 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:38.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.372+0000 7f7a60fb5640 10 librados: watch_flush enter 2024-03-25T17:39:38.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 10 librados: watch_flush exit 2024-03-25T17:39:38.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 20 client.4220.objecter shutdown clearing up homeless session... 2024-03-25T17:39:38.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 10 client.4220.objecter successfully canceled tick 2024-03-25T17:39:38.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 >> v1:172.21.15.44:6800/3257043132 conn(0x7f7a3c06e0c0 legacy=0x7f7a3c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:38.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 10 monclient: shutdown 2024-03-25T17:39:38.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:38.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 >> v1:172.21.15.44:6790/0 conn(0x7f7a5c1430c0 legacy=0x7f7a5c132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:38.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 shutdown_connections 2024-03-25T17:39:38.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 >> 172.21.15.44:0/2752274473 conn(0x7f7a5c0b7ea0 msgr2=0x7f7a5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:38.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 shutdown_connections 2024-03-25T17:39:38.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 1 -- 172.21.15.44:0/2752274473 wait complete. 2024-03-25T17:39:38.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:38.373+0000 7f7a60fb5640 1 librados: shutdown 2024-03-25T17:39:38.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:40.284 INFO:tasks.ceph.osd.5.smithi111.stderr:2024-03-25T17:39:40.280+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T17:39:40.421 INFO:tasks.ceph.osd.4.smithi111.stderr:2024-03-25T17:39:40.412+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T17:39:40.496 INFO:tasks.ceph.osd.6.smithi111.stderr:2024-03-25T17:39:40.493+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T17:39:40.682 INFO:tasks.ceph.osd.7.smithi111.stderr:2024-03-25T17:39:40.678+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T17:39:42.241 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-03-25T17:39:42.238+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T17:39:42.511 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-03-25T17:39:42.507+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T17:39:43.087 INFO:tasks.ceph.osd.3.smithi044.stderr:2024-03-25T17:39:43.082+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T17:39:43.236 INFO:tasks.ceph.osd.2.smithi044.stderr:2024-03-25T17:39:43.232+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T17:39:45.194 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:45.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.308+0000 7f51ee36d640 10 monclient: get_monmap_and_config 2024-03-25T17:39:45.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.308+0000 7f51ee36d640 10 monclient: build_initial_monmap 2024-03-25T17:39:45.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.308+0000 7f51ee36d640 10 monclient: monmap: 2024-03-25T17:39:45.308 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:39:45.309027+0000 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:39:45.309027+0000 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:45.309 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.309+0000 7f51ee36d640 1 Processor -- start 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.309+0000 7f51ee36d640 1 -- start start 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.309+0000 7f51ee36d640 10 monclient: init 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 10 monclient(hunting): picked mon.noname-c con 0x7f51e80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 10 monclient(hunting): picked mon.noname-b con 0x7f51e813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 10 monclient(hunting): picked mon.noname-a con 0x7f51e81430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:45.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e8142520 con 0x7f51e81430c0 2024-03-25T17:39:45.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e80b28a0 con 0x7f51e80a98f0 2024-03-25T17:39:45.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e80b2a30 con 0x7f51e813f710 2024-03-25T17:39:45.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:45.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.310+0000 7f51ee36d640 10 monclient(hunting): authenticate will time out at 5988.538114s 2024-03-25T17:39:45.318 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.317+0000 7f51e77fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f51e813f710 0x7f51e81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:38874/0 (socket says 172.21.15.44:38874) 2024-03-25T17:39:45.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.317+0000 7f51e77fe640 1 -- 172.21.15.44:0/2888171791 learned_addr learned my addr 172.21.15.44:0/2888171791 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:45.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.328+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333497306 0 0) 0x7f51e80b2a30 con 0x7f51e813f710 2024-03-25T17:39:45.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.328+0000 7f51e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.328+0000 7f51e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.328+0000 7f51e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.328+0000 7f51e6ffd640 10 monclient(hunting): my global_id is 4247 2024-03-25T17:39:45.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.328+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d0003520 con 0x7f51e813f710 2024-03-25T17:39:45.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.335+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085683372 0 0) 0x7f51e8142520 con 0x7f51e81430c0 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.335+0000 7f51e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.335+0000 7f51e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.335+0000 7f51e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.335+0000 7f51e6ffd640 10 monclient(hunting): my global_id is 4260 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.335+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e80b2a30 con 0x7f51e81430c0 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.340+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15452398 0 0) 0x7f51e80b28a0 con 0x7f51e80a98f0 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.340+0000 7f51e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.340+0000 7f51e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.340+0000 7f51e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.340+0000 7f51e6ffd640 10 monclient(hunting): my global_id is 4246 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.340+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e8142520 con 0x7f51e80a98f0 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013375010 0 0) 0x7f51e8142520 con 0x7f51e80a98f0 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 >> v1:172.21.15.44:6790/0 conn(0x7f51e813f710 legacy=0x7f51e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 >> v1:172.21.15.44:6789/0 conn(0x7f51e81430c0 legacy=0x7f51e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 1 monclient: found mon.noname-c 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e80b44c0 con 0x7f51e80a98f0 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f51d80015b0 con 0x7f51e80a98f0 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51e6ffd640 10 monclient: dump: 2024-03-25T17:39:45.358 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51ee36d640 5 monclient: authenticate success, global_id 4246 2024-03-25T17:39:45.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.355+0000 7f51ee36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.363+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51d8001710 con 0x7f51e80a98f0 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.363+0000 7f51e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.363+0000 7f51e6ffd640 1 -- 172.21.15.44:0/2888171791 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f51d8001b20 con 0x7f51e80a98f0 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.363+0000 7f51e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.363+0000 7f51e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.363+0000 7f51e6ffd640 10 monclient: dump: 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:45.364 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.363+0000 7f51ee36d640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:45.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.363+0000 7f51ee36d640 4 set_mon_vals no callback set 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.364+0000 7f51ee36d640 10 monclient: shutdown 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.364+0000 7f51ee36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.364+0000 7f51ee36d640 1 -- 172.21.15.44:0/2888171791 >> v1:172.21.15.111:6789/0 conn(0x7f51e80a98f0 legacy=0x7f51e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.364+0000 7f51ee36d640 1 -- 172.21.15.44:0/2888171791 shutdown_connections 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.364+0000 7f51ee36d640 1 -- 172.21.15.44:0/2888171791 >> 172.21.15.44:0/2888171791 conn(0x7f51e80b7ea0 msgr2=0x7f51e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.364+0000 7f51ee36d640 1 -- 172.21.15.44:0/2888171791 shutdown_connections 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.364+0000 7f51ee36d640 1 -- 172.21.15.44:0/2888171791 wait complete. 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.365+0000 7f51ee36d640 10 monclient: build_initial_monmap 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.365+0000 7f51ee36d640 10 monclient: monmap: 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:39:45.367 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.366+0000 7f51ee36d640 1 librados: starting msgr at 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.366+0000 7f51ee36d640 1 librados: starting objecter 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.366+0000 7f51ee36d640 1 Processor -- start 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.366+0000 7f51ee36d640 1 -- start start 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.366+0000 7f51ee36d640 1 librados: setting wanted keys 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.366+0000 7f51ee36d640 1 librados: calling monclient init 2024-03-25T17:39:45.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.366+0000 7f51ee36d640 10 monclient: init 2024-03-25T17:39:45.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:45.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:39:45.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 10 monclient(hunting): picked mon.noname-a con 0x7f51e80a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:45.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 10 monclient(hunting): picked mon.noname-c con 0x7f51e813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:45.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 10 monclient(hunting): picked mon.noname-b con 0x7f51e81430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:45.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e81d7780 con 0x7f51e80a98f0 2024-03-25T17:39:45.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e8137c50 con 0x7f51e81430c0 2024-03-25T17:39:45.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e81d41e0 con 0x7f51e813f710 2024-03-25T17:39:45.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:45.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.367+0000 7f51ee36d640 10 monclient(hunting): authenticate will time out at 5988.594940s 2024-03-25T17:39:45.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.389+0000 7f51ec90a640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f51e81430c0 0x7f51e8138710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:41132/0 (socket says 172.21.15.44:41132) 2024-03-25T17:39:45.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.389+0000 7f51ec90a640 1 -- 172.21.15.44:0/390735785 learned_addr learned my addr 172.21.15.44:0/390735785 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:45.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.391+0000 7f51e4ff9640 10 client.?.objecter ms_handle_connect 0x7f51e813f710 2024-03-25T17:39:45.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.391+0000 7f51e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:45.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.393+0000 7f51e4ff9640 10 client.?.objecter ms_handle_connect 0x7f51e81430c0 2024-03-25T17:39:45.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.393+0000 7f51e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:45.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.394+0000 7f51e4ff9640 10 client.?.objecter ms_handle_connect 0x7f51e80a98f0 2024-03-25T17:39:45.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.394+0000 7f51e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:45.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.400+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262824712 0 0) 0x7f51e81d41e0 con 0x7f51e813f710 2024-03-25T17:39:45.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.400+0000 7f51e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.400+0000 7f51e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.400+0000 7f51e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.400+0000 7f51e4ff9640 10 monclient(hunting): my global_id is 4250 2024-03-25T17:39:45.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.400+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d4004eb0 con 0x7f51e813f710 2024-03-25T17:39:45.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.405+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987484392 0 0) 0x7f51e8137c50 con 0x7f51e81430c0 2024-03-25T17:39:45.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.405+0000 7f51e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.405+0000 7f51e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.405+0000 7f51e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.405+0000 7f51e4ff9640 10 monclient(hunting): my global_id is 4249 2024-03-25T17:39:45.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.405+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e81d41e0 con 0x7f51e81430c0 2024-03-25T17:39:45.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.419+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328934956 0 0) 0x7f51e81d7780 con 0x7f51e80a98f0 2024-03-25T17:39:45.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.419+0000 7f51e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.419+0000 7f51e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.419+0000 7f51e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.419+0000 7f51e4ff9640 10 monclient(hunting): my global_id is 4263 2024-03-25T17:39:45.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.419+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e8137c50 con 0x7f51e80a98f0 2024-03-25T17:39:45.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.423+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042028373 0 0) 0x7f51d4004eb0 con 0x7f51e813f710 2024-03-25T17:39:45.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.423+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51e81d7780 con 0x7f51e813f710 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.423+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f51dc001950 con 0x7f51e813f710 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.423+0000 7f51e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.423+0000 7f51e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.423+0000 7f51e4ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:45.424 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.423+0000 7f51e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:45.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.430+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200220119 0 0) 0x7f51e81d41e0 con 0x7f51e81430c0 2024-03-25T17:39:45.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.431+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51d4004eb0 con 0x7f51e81430c0 2024-03-25T17:39:45.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.431+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f51e0001c10 con 0x7f51e81430c0 2024-03-25T17:39:45.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.431+0000 7f51e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.431+0000 7f51e4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:45.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.431+0000 7f51e4ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:45.431 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:45.431 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:45.432 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.432 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.432 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.432 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:45.432 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:45.432 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:45.432 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:45.432 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.438+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894156817 0 0) 0x7f51e8137c50 con 0x7f51e80a98f0 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.438+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51e81d41e0 con 0x7f51e80a98f0 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.438+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f51d8001a20 con 0x7f51e80a98f0 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.438+0000 7f51e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.438+0000 7f51e4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.438+0000 7f51e4ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:45.439 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:45.440 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:45.440 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:45.440 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.439+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2498789634 0 0) 0x7f51d4004eb0 con 0x7f51e81430c0 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.439+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 >> v1:172.21.15.44:6790/0 conn(0x7f51e813f710 legacy=0x7f51e81d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.439+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 >> v1:172.21.15.44:6789/0 conn(0x7f51e80a98f0 legacy=0x7f51e81d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.439+0000 7f51e4ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.439+0000 7f51e4ff9640 1 monclient: found mon.b 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.439+0000 7f51e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.439+0000 7f51e4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e81d4370 con 0x7f51e81430c0 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51e4ff9640 10 monclient: _finish_auth 0 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51e4ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:45.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 5 monclient: authenticate success, global_id 4249 2024-03-25T17:39:45.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 10 monclient: _renew_subs 2024-03-25T17:39:45.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:45.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e81d83b0 con 0x7f51e81430c0 2024-03-25T17:39:45.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:45.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 10 monclient: _renew_subs 2024-03-25T17:39:45.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:45.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e81d5080 con 0x7f51e81430c0 2024-03-25T17:39:45.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.440+0000 7f51ee36d640 1 librados: init done 2024-03-25T17:39:45.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.441+0000 7f51ee36d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:45.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.441+0000 7f51ee36d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:45.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.441+0000 7f51ee36d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:45.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.441+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e80b3ef0 con 0x7f51e81430c0 2024-03-25T17:39:45.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.723+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51e0003990 con 0x7f51e81430c0 2024-03-25T17:39:45.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.723+0000 7f51e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.723+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f51e0002100 con 0x7f51e81430c0 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.723+0000 7f51e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.723+0000 7f51e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.723+0000 7f51e4ff9640 10 monclient: dump: 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:45.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.723+0000 7f51e67fc640 4 set_mon_vals no callback set 2024-03-25T17:39:45.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.724+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f51e0003da0 con 0x7f51e81430c0 2024-03-25T17:39:45.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.725+0000 7f51e7fff640 10 monclient: get_auth_request con 0x7f51d406e160 auth_method 0 2024-03-25T17:39:45.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.725+0000 7f51e4ff9640 10 client.4249.objecter ms_handle_connect 0x7f51d406e160 2024-03-25T17:39:45.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.728+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f51e0054a90 con 0x7f51e81430c0 2024-03-25T17:39:45.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.728+0000 7f51e4ff9640 10 client.4249.objecter ms_dispatch 0x7f51e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:45.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.728+0000 7f51e4ff9640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:45.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.728+0000 7f51e4ff9640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:45.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.728+0000 7f51e4ff9640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-25T17:39:45.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.729+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51e00505a0 con 0x7f51e81430c0 2024-03-25T17:39:45.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.729+0000 7f51e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:45.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.729+0000 7f51e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:45.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.987+0000 7f51ee36d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:45.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.987+0000 7f51ee36d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:45.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.987+0000 7f51ee36d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:39:45.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:45.987+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51e81d5330 con 0x7f51e81430c0 2024-03-25T17:39:46.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.012+0000 7f51e4ff9640 1 -- 172.21.15.44:0/390735785 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f51e00524d0 con 0x7f51e81430c0 2024-03-25T17:39:46.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.012+0000 7f51e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:46.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.012+0000 7f51e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:46.013 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:39:46.013 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:46.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 10 librados: watch_flush enter 2024-03-25T17:39:46.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 10 librados: watch_flush exit 2024-03-25T17:39:46.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-25T17:39:46.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 10 client.4249.objecter successfully canceled tick 2024-03-25T17:39:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 >> v1:172.21.15.44:6800/3257043132 conn(0x7f51d406e160 legacy=0x7f51d4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 10 monclient: shutdown 2024-03-25T17:39:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 >> v1:172.21.15.111:6789/0 conn(0x7f51e81430c0 legacy=0x7f51e8138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 shutdown_connections 2024-03-25T17:39:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 >> 172.21.15.44:0/390735785 conn(0x7f51e80b7ea0 msgr2=0x7f51e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 shutdown_connections 2024-03-25T17:39:46.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 1 -- 172.21.15.44:0/390735785 wait complete. 2024-03-25T17:39:46.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:46.015+0000 7f51ee36d640 1 librados: shutdown 2024-03-25T17:39:46.028 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:52.834 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.950+0000 7f27054a8640 10 monclient: get_monmap_and_config 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.950+0000 7f27054a8640 10 monclient: build_initial_monmap 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.950+0000 7f27054a8640 10 monclient: monmap: 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:39:52.951574+0000 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:39:52.951574+0000 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:39:52.951 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.951+0000 7f27054a8640 1 Processor -- start 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.951+0000 7f27054a8640 1 -- start start 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.951+0000 7f27054a8640 10 monclient: init 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 10 monclient(hunting): picked mon.noname-b con 0x7f27000a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 10 monclient(hunting): picked mon.noname-a con 0x7f270013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 10 monclient(hunting): picked mon.noname-c con 0x7f27001430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700142520 con 0x7f270013f710 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000b28a0 con 0x7f27001430c0 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000b2a30 con 0x7f27000a98f0 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:52.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.952+0000 7f27054a8640 10 monclient(hunting): authenticate will time out at 5996.180222s 2024-03-25T17:39:52.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.962+0000 7f26fe7fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f270013f710 0x7f27001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:49860/0 (socket says 172.21.15.44:49860) 2024-03-25T17:39:52.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.962+0000 7f26fe7fc640 1 -- 172.21.15.44:0/419773655 learned_addr learned my addr 172.21.15.44:0/419773655 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:52.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.976+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58946763 0 0) 0x7f2700142520 con 0x7f270013f710 2024-03-25T17:39:52.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.976+0000 7f26fdffb640 10 monclient(hunting): _negotiate 2024-03-25T17:39:52.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.976+0000 7f26fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:52.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.976+0000 7f26fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:52.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.976+0000 7f26fdffb640 10 monclient(hunting): my global_id is 4266 2024-03-25T17:39:52.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.976+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e8003520 con 0x7f270013f710 2024-03-25T17:39:52.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.988+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833515818 0 0) 0x7f27000b28a0 con 0x7f27001430c0 2024-03-25T17:39:52.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.988+0000 7f26fdffb640 10 monclient(hunting): _negotiate 2024-03-25T17:39:52.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.988+0000 7f26fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:52.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.988+0000 7f26fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:52.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.988+0000 7f26fdffb640 10 monclient(hunting): my global_id is 4252 2024-03-25T17:39:52.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.988+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2700142520 con 0x7f27001430c0 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2768577219 0 0) 0x7f26e8003520 con 0x7f270013f710 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 >> v1:172.21.15.44:6790/0 conn(0x7f27000a98f0 legacy=0x7f27000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 >> v1:172.21.15.111:6789/0 conn(0x7f27001430c0 legacy=0x7f270014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 10 monclient: _finish_hunting 0 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 1 monclient: found mon.noname-a 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27000b44c0 con 0x7f270013f710 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 10 monclient: _finish_auth 0 2024-03-25T17:39:52.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 10 monclient: _check_auth_tickets 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f26f4001690 con 0x7f270013f710 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f26fdffb640 10 monclient: dump: 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:52.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f27054a8640 5 monclient: authenticate success, global_id 4266 2024-03-25T17:39:52.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:52.994+0000 7f27054a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.011+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26f40017f0 con 0x7f270013f710 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.011+0000 7f26fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.011+0000 7f26fdffb640 1 -- 172.21.15.44:0/419773655 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f26f4001c00 con 0x7f270013f710 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.011+0000 7f27054a8640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.011+0000 7f27054a8640 4 set_mon_vals no callback set 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.012+0000 7f26fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.012+0000 7f26fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.012+0000 7f26fdffb640 10 monclient: dump: 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:53.012 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:53.013 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.013 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.013 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.013 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:53.013 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:53.013 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:53.013 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:53.013 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:53.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.012+0000 7f27054a8640 10 monclient: shutdown 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.012+0000 7f27054a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.012+0000 7f27054a8640 1 -- 172.21.15.44:0/419773655 >> v1:172.21.15.44:6789/0 conn(0x7f270013f710 legacy=0x7f27001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.012+0000 7f27054a8640 1 -- 172.21.15.44:0/419773655 shutdown_connections 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.012+0000 7f27054a8640 1 -- 172.21.15.44:0/419773655 >> 172.21.15.44:0/419773655 conn(0x7f27000b7ea0 msgr2=0x7f27000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.013+0000 7f27054a8640 1 -- 172.21.15.44:0/419773655 shutdown_connections 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.013+0000 7f27054a8640 1 -- 172.21.15.44:0/419773655 wait complete. 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.013+0000 7f27054a8640 10 monclient: build_initial_monmap 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.013+0000 7f27054a8640 10 monclient: monmap: 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:39:53.015 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.014+0000 7f27054a8640 1 librados: starting msgr at 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.014+0000 7f27054a8640 1 librados: starting objecter 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.014+0000 7f27054a8640 1 Processor -- start 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.014+0000 7f27054a8640 1 -- start start 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.014+0000 7f27054a8640 1 librados: setting wanted keys 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.014+0000 7f27054a8640 1 librados: calling monclient init 2024-03-25T17:39:53.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.014+0000 7f27054a8640 10 monclient: init 2024-03-25T17:39:53.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:53.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:53.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 10 monclient(hunting): picked mon.noname-a con 0x7f27000a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:39:53.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 10 monclient(hunting): picked mon.noname-b con 0x7f270013f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:39:53.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 10 monclient(hunting): picked mon.noname-c con 0x7f27001430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:39:53.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700135550 con 0x7f27000a98f0 2024-03-25T17:39:53.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700136b50 con 0x7f270013f710 2024-03-25T17:39:53.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700136e30 con 0x7f27001430c0 2024-03-25T17:39:53.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:53.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.015+0000 7f27054a8640 10 monclient(hunting): authenticate will time out at 5996.242963s 2024-03-25T17:39:53.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.023+0000 7f26ff7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f27001430c0 0x7f27001e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:43758/0 (socket says 172.21.15.44:43758) 2024-03-25T17:39:53.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.023+0000 7f26ff7fe640 1 -- 172.21.15.44:0/2795228698 learned_addr learned my addr 172.21.15.44:0/2795228698 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:39:53.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.024+0000 7f26e77fe640 10 client.?.objecter ms_handle_connect 0x7f270013f710 2024-03-25T17:39:53.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.024+0000 7f26e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:53.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.026+0000 7f26e77fe640 10 client.?.objecter ms_handle_connect 0x7f27001430c0 2024-03-25T17:39:53.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.026+0000 7f26e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:53.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.032+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200043365 0 0) 0x7f2700136b50 con 0x7f270013f710 2024-03-25T17:39:53.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.032+0000 7f26e77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.032+0000 7f26e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.032+0000 7f26e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.032+0000 7f26e77fe640 10 monclient(hunting): my global_id is 4255 2024-03-25T17:39:53.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.032+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ec004eb0 con 0x7f270013f710 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.036+0000 7f26e77fe640 10 client.?.objecter ms_handle_connect 0x7f27000a98f0 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.036+0000 7f26e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.038+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315232994 0 0) 0x7f2700136e30 con 0x7f27001430c0 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.038+0000 7f26e77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.038+0000 7f26e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.038+0000 7f26e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.038+0000 7f26e77fe640 10 monclient(hunting): my global_id is 4256 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.038+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2700136b50 con 0x7f27001430c0 2024-03-25T17:39:53.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.042+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518819875 0 0) 0x7f2700135550 con 0x7f27000a98f0 2024-03-25T17:39:53.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.042+0000 7f26e77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.042+0000 7f26e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.042+0000 7f26e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.042+0000 7f26e77fe640 10 monclient(hunting): my global_id is 4269 2024-03-25T17:39:53.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.042+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2700136e30 con 0x7f27000a98f0 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.044+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849028732 0 0) 0x7f26ec004eb0 con 0x7f270013f710 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.044+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2700135550 con 0x7f270013f710 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.044+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f26f4001bc0 con 0x7f270013f710 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.044+0000 7f26e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.044+0000 7f26e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.044+0000 7f26e77fe640 10 monclient(hunting): dump: 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.045 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:53.046 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:53.046 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:53.046 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:53.046 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:53.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.044+0000 7f26e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.051+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3478240932 0 0) 0x7f2700136e30 con 0x7f27000a98f0 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.051+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26ec004eb0 con 0x7f27000a98f0 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.051+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f26f0001860 con 0x7f27000a98f0 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.051+0000 7f26e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.051+0000 7f26e77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.051+0000 7f26e77fe640 10 monclient(hunting): dump: 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.052 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.053 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:53.053 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:53.053 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:53.053 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:53.053 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:53.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.058+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3983003776 0 0) 0x7f2700136b50 con 0x7f27001430c0 2024-03-25T17:39:53.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.059+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2700136e30 con 0x7f27001430c0 2024-03-25T17:39:53.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.059+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f26f8001c10 con 0x7f27001430c0 2024-03-25T17:39:53.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.059+0000 7f26e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.059+0000 7f26e77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:53.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.059+0000 7f26e77fe640 10 monclient(hunting): dump: 2024-03-25T17:39:53.059 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:53.059 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 914615276 0 0) 0x7f26ec004eb0 con 0x7f27000a98f0 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 >> v1:172.21.15.44:6790/0 conn(0x7f27001430c0 legacy=0x7f27001e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 >> v1:172.21.15.111:6789/0 conn(0x7f270013f710 legacy=0x7f27001375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 1 monclient: found mon.a 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27001357f0 con 0x7f27000a98f0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f26e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f27054a8640 5 monclient: authenticate success, global_id 4269 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f27054a8640 10 monclient: _renew_subs 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f27054a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.067+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2700137cb0 con 0x7f27000a98f0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.068+0000 7f27054a8640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.068+0000 7f27054a8640 10 monclient: _renew_subs 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.068+0000 7f27054a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.068+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27001352e0 con 0x7f27000a98f0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.068+0000 7f27054a8640 1 librados: init done 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.069+0000 7f27054a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.069+0000 7f27054a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.069+0000 7f27054a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.069+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2700074cc0 con 0x7f27000a98f0 2024-03-25T17:39:53.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.349+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26f0001d30 con 0x7f27000a98f0 2024-03-25T17:39:53.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.349+0000 7f26e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:53.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.349+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f26f0003790 con 0x7f27000a98f0 2024-03-25T17:39:53.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.349+0000 7f26e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.349+0000 7f26e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:53.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.349+0000 7f26e77fe640 10 monclient: dump: 2024-03-25T17:39:53.350 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:39:53.350 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.349+0000 7f26fd7fa640 4 set_mon_vals no callback set 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.350+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f26f00197e0 con 0x7f27000a98f0 2024-03-25T17:39:53.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.350+0000 7f26fe7fc640 10 monclient: get_auth_request con 0x7f26ec06e160 auth_method 0 2024-03-25T17:39:53.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.350+0000 7f26e77fe640 10 client.4269.objecter ms_handle_connect 0x7f26ec06e160 2024-03-25T17:39:53.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.355+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f26f0053480 con 0x7f27000a98f0 2024-03-25T17:39:53.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.355+0000 7f26e77fe640 10 client.4269.objecter ms_dispatch 0x7f2700000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:53.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.355+0000 7f26e77fe640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:53.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.355+0000 7f26e77fe640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:53.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.355+0000 7f26e77fe640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-25T17:39:53.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.355+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26f004edc0 con 0x7f27000a98f0 2024-03-25T17:39:53.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.355+0000 7f26e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:53.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.355+0000 7f26e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:53.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.596+0000 7f27054a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:53.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.596+0000 7f27054a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:53.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.596+0000 7f27054a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:39:53.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.596+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2700135980 con 0x7f27000a98f0 2024-03-25T17:39:53.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.629+0000 7f26e77fe640 1 -- 172.21.15.44:0/2795228698 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f26f0050ec0 con 0x7f27000a98f0 2024-03-25T17:39:53.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.629+0000 7f26e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:53.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.629+0000 7f26e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:53.629 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:39:53.630 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:53.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 10 librados: watch_flush enter 2024-03-25T17:39:53.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 10 librados: watch_flush exit 2024-03-25T17:39:53.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-25T17:39:53.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 10 client.4269.objecter successfully canceled tick 2024-03-25T17:39:53.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 >> v1:172.21.15.44:6800/3257043132 conn(0x7f26ec06e160 legacy=0x7f26ec070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 10 monclient: shutdown 2024-03-25T17:39:53.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:53.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 >> v1:172.21.15.44:6789/0 conn(0x7f27000a98f0 legacy=0x7f270013b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 shutdown_connections 2024-03-25T17:39:53.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 >> 172.21.15.44:0/2795228698 conn(0x7f27000b7ea0 msgr2=0x7f2700074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:53.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 shutdown_connections 2024-03-25T17:39:53.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 1 -- 172.21.15.44:0/2795228698 wait complete. 2024-03-25T17:39:53.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:39:53.631+0000 7f27054a8640 1 librados: shutdown 2024-03-25T17:39:53.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:57.155 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:14.510 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:57.215 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:14.571 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:57.244 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:14.600 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:57.319 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:14.674 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:57.355 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:14.675 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:57.417 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:14.737 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:57.457 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:14.813 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:57.568 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:14.924 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:57.632 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:14.952 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:00.451 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:00.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.565+0000 7fea8448d640 10 monclient: get_monmap_and_config 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.565+0000 7fea8448d640 10 monclient: build_initial_monmap 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.565+0000 7fea8448d640 10 monclient: monmap: 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:40:00.566535+0000 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:40:00.566535+0000 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:40:00.566 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.566+0000 7fea8448d640 1 Processor -- start 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.566+0000 7fea8448d640 1 -- start start 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.566+0000 7fea8448d640 10 monclient: init 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 10 monclient(hunting): picked mon.noname-a con 0x7fea7c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 10 monclient(hunting): picked mon.noname-b con 0x7fea7c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 10 monclient(hunting): picked mon.noname-c con 0x7fea7c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c142520 con 0x7fea7c0a98f0 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c0b28a0 con 0x7fea7c1430c0 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c0b2a30 con 0x7fea7c13f710 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:00.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.567+0000 7fea8448d640 10 monclient(hunting): authenticate will time out at 6003.794975s 2024-03-25T17:40:00.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.574+0000 7fea81a28640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fea7c13f710 0x7fea7c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:47742/0 (socket says 172.21.15.44:47742) 2024-03-25T17:40:00.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.574+0000 7fea81a28640 1 -- 172.21.15.44:0/212376581 learned_addr learned my addr 172.21.15.44:0/212376581 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:00.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.601+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074667364 0 0) 0x7fea7c0b2a30 con 0x7fea7c13f710 2024-03-25T17:40:00.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.601+0000 7fea81227640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.601+0000 7fea81227640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.601+0000 7fea81227640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.601+0000 7fea81227640 10 monclient(hunting): my global_id is 4259 2024-03-25T17:40:00.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.601+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea68003520 con 0x7fea7c13f710 2024-03-25T17:40:00.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.603+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505860946 0 0) 0x7fea7c142520 con 0x7fea7c0a98f0 2024-03-25T17:40:00.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.603+0000 7fea81227640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.603+0000 7fea81227640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.603+0000 7fea81227640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.603+0000 7fea81227640 10 monclient(hunting): my global_id is 4272 2024-03-25T17:40:00.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.603+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c0b2a30 con 0x7fea7c0a98f0 2024-03-25T17:40:00.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.608+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098247490 0 0) 0x7fea7c0b28a0 con 0x7fea7c1430c0 2024-03-25T17:40:00.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.608+0000 7fea81227640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.608+0000 7fea81227640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.608+0000 7fea81227640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.608+0000 7fea81227640 10 monclient(hunting): my global_id is 4258 2024-03-25T17:40:00.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.608+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c142520 con 0x7fea7c1430c0 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1809150196 0 0) 0x7fea68003520 con 0x7fea7c13f710 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 >> v1:172.21.15.111:6789/0 conn(0x7fea7c1430c0 legacy=0x7fea7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 >> v1:172.21.15.44:6789/0 conn(0x7fea7c0a98f0 legacy=0x7fea7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 10 monclient: _finish_hunting 0 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 1 monclient: found mon.noname-b 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c0b44c0 con 0x7fea7c13f710 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 10 monclient: _finish_auth 0 2024-03-25T17:40:00.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 10 monclient: _check_auth_tickets 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fea6c001690 con 0x7fea7c13f710 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea81227640 10 monclient: dump: 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:00.622 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:00.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea8448d640 5 monclient: authenticate success, global_id 4259 2024-03-25T17:40:00.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.620+0000 7fea8448d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:00.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.640+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea6c0017f0 con 0x7fea7c13f710 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.640+0000 7fea81227640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.640+0000 7fea81227640 1 -- 172.21.15.44:0/212376581 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fea6c001c00 con 0x7fea7c13f710 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.640+0000 7fea8448d640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.640+0000 7fea8448d640 4 set_mon_vals no callback set 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.640+0000 7fea81227640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.640+0000 7fea81227640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.640+0000 7fea81227640 10 monclient: dump: 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:00.641 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:00.642 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:00.642 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:00.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.641+0000 7fea8448d640 10 monclient: shutdown 2024-03-25T17:40:00.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.641+0000 7fea8448d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:00.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.641+0000 7fea8448d640 1 -- 172.21.15.44:0/212376581 >> v1:172.21.15.44:6790/0 conn(0x7fea7c13f710 legacy=0x7fea7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:00.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.641+0000 7fea8448d640 1 -- 172.21.15.44:0/212376581 shutdown_connections 2024-03-25T17:40:00.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.641+0000 7fea8448d640 1 -- 172.21.15.44:0/212376581 >> 172.21.15.44:0/212376581 conn(0x7fea7c0b7ea0 msgr2=0x7fea7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:00.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.641+0000 7fea8448d640 1 -- 172.21.15.44:0/212376581 shutdown_connections 2024-03-25T17:40:00.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.641+0000 7fea8448d640 1 -- 172.21.15.44:0/212376581 wait complete. 2024-03-25T17:40:00.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.642+0000 7fea8448d640 10 monclient: build_initial_monmap 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.642+0000 7fea8448d640 10 monclient: monmap: 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.642+0000 7fea8448d640 1 librados: starting msgr at 2024-03-25T17:40:00.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 1 librados: starting objecter 2024-03-25T17:40:00.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 1 Processor -- start 2024-03-25T17:40:00.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 1 -- start start 2024-03-25T17:40:00.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 1 librados: setting wanted keys 2024-03-25T17:40:00.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 1 librados: calling monclient init 2024-03-25T17:40:00.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 10 monclient: init 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 10 monclient(hunting): picked mon.noname-b con 0x7fea7c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.643+0000 7fea8448d640 10 monclient(hunting): picked mon.noname-c con 0x7fea7c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.644+0000 7fea8448d640 10 monclient(hunting): picked mon.noname-a con 0x7fea7c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.644+0000 7fea8448d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c0b1870 con 0x7fea7c1430c0 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.644+0000 7fea8448d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c0ac910 con 0x7fea7c0a98f0 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.644+0000 7fea8448d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c0abb00 con 0x7fea7c13f710 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.644+0000 7fea8448d640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:00.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.644+0000 7fea8448d640 10 monclient(hunting): authenticate will time out at 6003.871791s 2024-03-25T17:40:00.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.662+0000 7fea81a28640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fea7c13f710 0x7fea7c0ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:47802/0 (socket says 172.21.15.44:47802) 2024-03-25T17:40:00.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.662+0000 7fea81a28640 1 -- 172.21.15.44:0/2389583822 learned_addr learned my addr 172.21.15.44:0/2389583822 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:00.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.662+0000 7fea72ffd640 10 client.?.objecter ms_handle_connect 0x7fea7c13f710 2024-03-25T17:40:00.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.663+0000 7fea72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:00.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.667+0000 7fea72ffd640 10 client.?.objecter ms_handle_connect 0x7fea7c0a98f0 2024-03-25T17:40:00.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.667+0000 7fea72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:00.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.674+0000 7fea72ffd640 10 client.?.objecter ms_handle_connect 0x7fea7c1430c0 2024-03-25T17:40:00.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.674+0000 7fea72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:00.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.675+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814202530 0 0) 0x7fea7c0ac910 con 0x7fea7c0a98f0 2024-03-25T17:40:00.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.675+0000 7fea72ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.675+0000 7fea72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.675+0000 7fea72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.675+0000 7fea72ffd640 10 monclient(hunting): my global_id is 4261 2024-03-25T17:40:00.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.675+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea60004eb0 con 0x7fea7c0a98f0 2024-03-25T17:40:00.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.679+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929596780 0 0) 0x7fea7c0abb00 con 0x7fea7c13f710 2024-03-25T17:40:00.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.679+0000 7fea72ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.679+0000 7fea72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.679+0000 7fea72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.679+0000 7fea72ffd640 10 monclient(hunting): my global_id is 4262 2024-03-25T17:40:00.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.679+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c0ac910 con 0x7fea7c13f710 2024-03-25T17:40:00.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.690+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376661692 0 0) 0x7fea7c142520 con 0x7fea7c1430c0 2024-03-25T17:40:00.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.690+0000 7fea72ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.690+0000 7fea72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.690+0000 7fea72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.690+0000 7fea72ffd640 10 monclient(hunting): my global_id is 4275 2024-03-25T17:40:00.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.690+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c0abb00 con 0x7fea7c1430c0 2024-03-25T17:40:00.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.692+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2066577186 0 0) 0x7fea60004eb0 con 0x7fea7c0a98f0 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.692+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7c142520 con 0x7fea7c0a98f0 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.692+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fea78001970 con 0x7fea7c0a98f0 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.692+0000 7fea72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.692+0000 7fea72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.692+0000 7fea72ffd640 10 monclient(hunting): dump: 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:00.693 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:00.694 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:00.694 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:00.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.692+0000 7fea72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.703+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1719467721 0 0) 0x7fea7c0ac910 con 0x7fea7c13f710 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.703+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea60004eb0 con 0x7fea7c13f710 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.703+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fea6c001b40 con 0x7fea7c13f710 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.703+0000 7fea72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.703+0000 7fea72ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.703+0000 7fea72ffd640 10 monclient(hunting): dump: 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.704 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.705 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.705 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:00.705 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:00.705 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:00.705 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:00.705 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.706+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 919265885 0 0) 0x7fea7c0abb00 con 0x7fea7c1430c0 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.706+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7c0ac910 con 0x7fea7c1430c0 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.706+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fea74001a30 con 0x7fea7c1430c0 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.706+0000 7fea72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.706+0000 7fea72ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.706+0000 7fea72ffd640 10 monclient(hunting): dump: 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.707 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.708 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.708 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:00.708 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:00.708 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:00.708 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:00.708 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3156703904 0 0) 0x7fea7c142520 con 0x7fea7c0a98f0 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 >> v1:172.21.15.44:6790/0 conn(0x7fea7c13f710 legacy=0x7fea7c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 >> v1:172.21.15.44:6789/0 conn(0x7fea7c1430c0 legacy=0x7fea7c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 1 monclient: found mon.b 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c0abde0 con 0x7fea7c0a98f0 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 10 monclient: _finish_auth 0 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 5 monclient: authenticate success, global_id 4261 2024-03-25T17:40:00.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 10 monclient: _renew_subs 2024-03-25T17:40:00.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:00.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7c138f40 con 0x7fea7c0a98f0 2024-03-25T17:40:00.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:00.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 10 monclient: _renew_subs 2024-03-25T17:40:00.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:00.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7c1363f0 con 0x7fea7c0a98f0 2024-03-25T17:40:00.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.712+0000 7fea8448d640 1 librados: init done 2024-03-25T17:40:00.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.713+0000 7fea8448d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:00.715 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.713+0000 7fea8448d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:00.715 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.713+0000 7fea8448d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:00.715 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.713+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7c0627b0 con 0x7fea7c0a98f0 2024-03-25T17:40:00.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.997+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea780035f0 con 0x7fea7c0a98f0 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.997+0000 7fea72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.997+0000 7fea80a26640 4 set_mon_vals no callback set 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.997+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fea78003900 con 0x7fea7c0a98f0 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.997+0000 7fea72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.997+0000 7fea72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.997+0000 7fea72ffd640 10 monclient: dump: 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:00.998 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:00.999 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:00.999 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:00.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.998+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fea780198d0 con 0x7fea7c0a98f0 2024-03-25T17:40:00.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.999+0000 7fea81a28640 10 monclient: get_auth_request con 0x7fea6006e0f0 auth_method 0 2024-03-25T17:40:00.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:00.999+0000 7fea72ffd640 10 client.4261.objecter ms_handle_connect 0x7fea6006e0f0 2024-03-25T17:40:01.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.000+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fea78053570 con 0x7fea7c0a98f0 2024-03-25T17:40:01.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.000+0000 7fea72ffd640 10 client.4261.objecter ms_dispatch 0x7fea7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:40:01.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.000+0000 7fea72ffd640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:40:01.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.000+0000 7fea72ffd640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:40:01.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.000+0000 7fea72ffd640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T17:40:01.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.003+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea7804f080 con 0x7fea7c0a98f0 2024-03-25T17:40:01.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.003+0000 7fea72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:01.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.003+0000 7fea72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:01.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.258+0000 7fea8448d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:01.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.258+0000 7fea8448d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:01.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.258+0000 7fea8448d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:01.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.258+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea7c1434a0 con 0x7fea7c0a98f0 2024-03-25T17:40:01.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.361+0000 7fea72ffd640 1 -- 172.21.15.44:0/2389583822 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fea78050fb0 con 0x7fea7c0a98f0 2024-03-25T17:40:01.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.361+0000 7fea72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:01.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.361+0000 7fea72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:01.362 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:40:01.362 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 10 librados: watch_flush enter 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 10 librados: watch_flush exit 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 10 client.4261.objecter successfully canceled tick 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 >> v1:172.21.15.44:6800/3257043132 conn(0x7fea6006e0f0 legacy=0x7fea600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 10 monclient: shutdown 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 >> v1:172.21.15.111:6789/0 conn(0x7fea7c0a98f0 legacy=0x7fea7c138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 shutdown_connections 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 >> 172.21.15.44:0/2389583822 conn(0x7fea7c0b7ea0 msgr2=0x7fea7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:01.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 shutdown_connections 2024-03-25T17:40:01.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 1 -- 172.21.15.44:0/2389583822 wait complete. 2024-03-25T17:40:01.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:01.364+0000 7fea8448d640 1 librados: shutdown 2024-03-25T17:40:01.377 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:08.182 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:08.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.300+0000 7f81d207b640 10 monclient: get_monmap_and_config 2024-03-25T17:40:08.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.300+0000 7f81d207b640 10 monclient: build_initial_monmap 2024-03-25T17:40:08.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.300+0000 7f81d207b640 10 monclient: monmap: 2024-03-25T17:40:08.301 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:08.301 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:08.301 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:40:08.301884+0000 2024-03-25T17:40:08.302 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:40:08.301884+0000 2024-03-25T17:40:08.302 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:08.302 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:08.302 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:08.302 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:40:08.302 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:40:08.302 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:08.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.301+0000 7f81d207b640 1 Processor -- start 2024-03-25T17:40:08.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.301+0000 7f81d207b640 1 -- start start 2024-03-25T17:40:08.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.301+0000 7f81d207b640 10 monclient: init 2024-03-25T17:40:08.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:08.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:40:08.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 10 monclient(hunting): picked mon.noname-b con 0x7f81cc0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:08.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 10 monclient(hunting): picked mon.noname-c con 0x7f81cc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:08.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 10 monclient(hunting): picked mon.noname-a con 0x7f81cc1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:08.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc142520 con 0x7f81cc1430c0 2024-03-25T17:40:08.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0b28a0 con 0x7f81cc13f710 2024-03-25T17:40:08.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0b2a30 con 0x7f81cc0a98f0 2024-03-25T17:40:08.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:08.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.302+0000 7f81d207b640 10 monclient(hunting): authenticate will time out at 6011.530615s 2024-03-25T17:40:08.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.310+0000 7f81cb7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f81cc0a98f0 0x7f81cc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:52404/0 (socket says 172.21.15.44:52404) 2024-03-25T17:40:08.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.310+0000 7f81cb7fe640 1 -- 172.21.15.44:0/2435911211 learned_addr learned my addr 172.21.15.44:0/2435911211 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:08.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.331+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557905477 0 0) 0x7f81cc0b28a0 con 0x7f81cc13f710 2024-03-25T17:40:08.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.331+0000 7f81ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.331+0000 7f81ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.331+0000 7f81ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.331+0000 7f81ca7fc640 10 monclient(hunting): my global_id is 4264 2024-03-25T17:40:08.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.331+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b4003520 con 0x7f81cc13f710 2024-03-25T17:40:08.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.333+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568470917 0 0) 0x7f81cc0b2a30 con 0x7f81cc0a98f0 2024-03-25T17:40:08.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.333+0000 7f81ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.333+0000 7f81ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.333+0000 7f81ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.333+0000 7f81ca7fc640 10 monclient(hunting): my global_id is 4265 2024-03-25T17:40:08.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.333+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc0b28a0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651713101 0 0) 0x7f81cc142520 con 0x7f81cc1430c0 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc0b2a30 con 0x7f81cc1430c0 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1370244182 0 0) 0x7f81b4003520 con 0x7f81cc13f710 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 >> v1:172.21.15.44:6790/0 conn(0x7f81cc0a98f0 legacy=0x7f81cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 >> v1:172.21.15.44:6789/0 conn(0x7f81cc1430c0 legacy=0x7f81cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 1 monclient: found mon.noname-c 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.340+0000 7f81ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:40:08.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc0b44c0 con 0x7f81cc13f710 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 10 monclient: _finish_auth 0 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f81c0001690 con 0x7f81cc13f710 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81ca7fc640 10 monclient: dump: 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:08.342 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:08.343 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:08.343 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:08.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81d207b640 5 monclient: authenticate success, global_id 4264 2024-03-25T17:40:08.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.341+0000 7f81d207b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:08.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.349+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81c00017f0 con 0x7f81cc13f710 2024-03-25T17:40:08.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.349+0000 7f81ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:08.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.349+0000 7f81ca7fc640 1 -- 172.21.15.44:0/2435911211 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f81c0001c00 con 0x7f81cc13f710 2024-03-25T17:40:08.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.349+0000 7f81ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.349+0000 7f81ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.349+0000 7f81ca7fc640 10 monclient: dump: 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.349+0000 7f81d207b640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:08.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.349+0000 7f81d207b640 4 set_mon_vals no callback set 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.350+0000 7f81d207b640 10 monclient: shutdown 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.350+0000 7f81d207b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.350+0000 7f81d207b640 1 -- 172.21.15.44:0/2435911211 >> v1:172.21.15.111:6789/0 conn(0x7f81cc13f710 legacy=0x7f81cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.350+0000 7f81d207b640 1 -- 172.21.15.44:0/2435911211 shutdown_connections 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.350+0000 7f81d207b640 1 -- 172.21.15.44:0/2435911211 >> 172.21.15.44:0/2435911211 conn(0x7f81cc0b7ea0 msgr2=0x7f81cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.350+0000 7f81d207b640 1 -- 172.21.15.44:0/2435911211 shutdown_connections 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.350+0000 7f81d207b640 1 -- 172.21.15.44:0/2435911211 wait complete. 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.351+0000 7f81d207b640 10 monclient: build_initial_monmap 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.351+0000 7f81d207b640 10 monclient: monmap: 2024-03-25T17:40:08.352 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.351+0000 7f81d207b640 1 librados: starting msgr at 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.351+0000 7f81d207b640 1 librados: starting objecter 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.351+0000 7f81d207b640 1 Processor -- start 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.352+0000 7f81d207b640 1 -- start start 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.352+0000 7f81d207b640 1 librados: setting wanted keys 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.352+0000 7f81d207b640 1 librados: calling monclient init 2024-03-25T17:40:08.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.352+0000 7f81d207b640 10 monclient: init 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 10 monclient(hunting): picked mon.noname-c con 0x7f81cc0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 10 monclient(hunting): picked mon.noname-b con 0x7f81cc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 10 monclient(hunting): picked mon.noname-a con 0x7f81cc1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc1d7490 con 0x7f81cc1430c0 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc1d76c0 con 0x7f81cc13f710 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc1d82f0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:08.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.353+0000 7f81d207b640 10 monclient(hunting): authenticate will time out at 6011.581393s 2024-03-25T17:40:08.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.366+0000 7f81caffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f81cc13f710 0x7f81cc1383e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:36614/0 (socket says 172.21.15.44:36614) 2024-03-25T17:40:08.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.366+0000 7f81caffd640 1 -- 172.21.15.44:0/294014811 learned_addr learned my addr 172.21.15.44:0/294014811 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:08.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.366+0000 7f81b3fff640 10 client.?.objecter ms_handle_connect 0x7f81cc13f710 2024-03-25T17:40:08.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.366+0000 7f81b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:08.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.371+0000 7f81b3fff640 10 client.?.objecter ms_handle_connect 0x7f81cc0a98f0 2024-03-25T17:40:08.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.371+0000 7f81b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:08.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.380+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704997711 0 0) 0x7f81cc1d82f0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.380+0000 7f81b3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.380+0000 7f81b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.380+0000 7f81b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.380+0000 7f81b3fff640 10 monclient(hunting): my global_id is 4268 2024-03-25T17:40:08.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.380+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b8004eb0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.385+0000 7f81b3fff640 10 client.?.objecter ms_handle_connect 0x7f81cc1430c0 2024-03-25T17:40:08.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.385+0000 7f81b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:08.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.389+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378561555 0 0) 0x7f81cc1d76c0 con 0x7f81cc13f710 2024-03-25T17:40:08.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.389+0000 7f81b3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.389+0000 7f81b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.389+0000 7f81b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.389+0000 7f81b3fff640 10 monclient(hunting): my global_id is 4267 2024-03-25T17:40:08.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.389+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc1d82f0 con 0x7f81cc13f710 2024-03-25T17:40:08.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.405+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791032977 0 0) 0x7f81cc1d7490 con 0x7f81cc1430c0 2024-03-25T17:40:08.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.405+0000 7f81b3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.405+0000 7f81b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.405+0000 7f81b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.405+0000 7f81b3fff640 10 monclient(hunting): my global_id is 4281 2024-03-25T17:40:08.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.405+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc1d76c0 con 0x7f81cc1430c0 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.409+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1607550198 0 0) 0x7f81b8004eb0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.409+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81cc1d7490 con 0x7f81cc0a98f0 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.409+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f81bc001970 con 0x7f81cc0a98f0 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.409+0000 7f81b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.409+0000 7f81b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.409+0000 7f81b3fff640 10 monclient(hunting): dump: 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.410 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:08.411 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:08.411 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:08.411 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:08.411 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:08.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.409+0000 7f81b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:08.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.417+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2734396620 0 0) 0x7f81cc1d82f0 con 0x7f81cc13f710 2024-03-25T17:40:08.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.417+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81b8004eb0 con 0x7f81cc13f710 2024-03-25T17:40:08.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.417+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f81c0001b40 con 0x7f81cc13f710 2024-03-25T17:40:08.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.417+0000 7f81b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.417+0000 7f81b3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.417+0000 7f81b3fff640 10 monclient(hunting): dump: 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:08.418 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:08.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3617230441 0 0) 0x7f81cc1d7490 con 0x7f81cc0a98f0 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 >> v1:172.21.15.111:6789/0 conn(0x7f81cc13f710 legacy=0x7f81cc1383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 >> v1:172.21.15.44:6789/0 conn(0x7f81cc1430c0 legacy=0x7f81cc133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 1 monclient: found mon.c 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc138af0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 10 monclient: _finish_auth 0 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 5 monclient: authenticate success, global_id 4268 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 10 monclient: _renew_subs 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc1376d0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:08.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 10 monclient: _renew_subs 2024-03-25T17:40:08.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:08.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc134fc0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.427+0000 7f81d207b640 1 librados: init done 2024-03-25T17:40:08.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.428+0000 7f81d207b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:08.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.428+0000 7f81d207b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:08.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.428+0000 7f81d207b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:08.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.428+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc1d7850 con 0x7f81cc0a98f0 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.768+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81bc0047e0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.768+0000 7f81b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.768+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f81bc004af0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.768+0000 7f81b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.768+0000 7f81b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.768+0000 7f81b3fff640 10 monclient: dump: 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:08.769 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:08.770 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:08.770 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:08.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.768+0000 7f81c9ffb640 4 set_mon_vals no callback set 2024-03-25T17:40:08.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.769+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f81bc01aac0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.770+0000 7f81caffd640 10 monclient: get_auth_request con 0x7f81b806df20 auth_method 0 2024-03-25T17:40:08.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.770+0000 7f81b3fff640 10 client.4268.objecter ms_handle_connect 0x7f81b806df20 2024-03-25T17:40:08.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.775+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f81bc0547a0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.775+0000 7f81b3fff640 10 client.4268.objecter ms_dispatch 0x7f81cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:40:08.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.775+0000 7f81b3fff640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:40:08.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.775+0000 7f81b3fff640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:40:08.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.775+0000 7f81b3fff640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-25T17:40:08.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.776+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81bc0502b0 con 0x7f81cc0a98f0 2024-03-25T17:40:08.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.776+0000 7f81b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:08.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:08.776+0000 7f81b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:09.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.028+0000 7f81d207b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:09.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.028+0000 7f81d207b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:09.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.028+0000 7f81d207b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:09.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.028+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81cc0627b0 con 0x7f81cc0a98f0 2024-03-25T17:40:09.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.066+0000 7f81b3fff640 1 -- 172.21.15.44:0/294014811 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f81bc0521e0 con 0x7f81cc0a98f0 2024-03-25T17:40:09.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.066+0000 7f81b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:09.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.066+0000 7f81b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:09.067 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:40:09.067 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 10 librados: watch_flush enter 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 10 librados: watch_flush exit 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 10 client.4268.objecter successfully canceled tick 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 >> v1:172.21.15.44:6800/3257043132 conn(0x7f81b806df20 legacy=0x7f81b80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 10 monclient: shutdown 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 >> v1:172.21.15.44:6790/0 conn(0x7f81cc0a98f0 legacy=0x7f81cc1d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 shutdown_connections 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 >> 172.21.15.44:0/294014811 conn(0x7f81cc0b7ea0 msgr2=0x7f81cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 shutdown_connections 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 1 -- 172.21.15.44:0/294014811 wait complete. 2024-03-25T17:40:09.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:09.069+0000 7f81d207b640 1 librados: shutdown 2024-03-25T17:40:09.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:15.887 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.003+0000 7efc99b90640 10 monclient: get_monmap_and_config 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.003+0000 7efc99b90640 10 monclient: build_initial_monmap 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.003+0000 7efc99b90640 10 monclient: monmap: 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:40:16.005463+0000 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:40:16.005463+0000 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:16.005 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:16.006 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:16.006 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:40:16.006 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:40:16.006 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:16.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.004+0000 7efc99b90640 1 Processor -- start 2024-03-25T17:40:16.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.004+0000 7efc99b90640 1 -- start start 2024-03-25T17:40:16.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.004+0000 7efc99b90640 10 monclient: init 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 10 monclient(hunting): picked mon.noname-a con 0x7efc940a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 10 monclient(hunting): picked mon.noname-b con 0x7efc9413f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 10 monclient(hunting): picked mon.noname-c con 0x7efc941430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc94142520 con 0x7efc940a98f0 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc940b28a0 con 0x7efc941430c0 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc940b2a30 con 0x7efc9413f710 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:16.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.005+0000 7efc99b90640 10 monclient(hunting): authenticate will time out at 6019.234187s 2024-03-25T17:40:16.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.007+0000 7efc93fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efc941430c0 0x7efc9414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60282/0 (socket says 172.21.15.44:60282) 2024-03-25T17:40:16.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.007+0000 7efc93fff640 1 -- 172.21.15.44:0/2134173979 learned_addr learned my addr 172.21.15.44:0/2134173979 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:16.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.102+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259168872 0 0) 0x7efc94142520 con 0x7efc940a98f0 2024-03-25T17:40:16.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.102+0000 7efc927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:16.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.102+0000 7efc927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:16.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.102+0000 7efc927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:16.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.102+0000 7efc927fc640 10 monclient(hunting): my global_id is 4284 2024-03-25T17:40:16.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.102+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7c003520 con 0x7efc940a98f0 2024-03-25T17:40:16.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1958013062 0 0) 0x7efc7c003520 con 0x7efc940a98f0 2024-03-25T17:40:16.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 >> v1:172.21.15.44:6790/0 conn(0x7efc9413f710 legacy=0x7efc941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 >> v1:172.21.15.111:6789/0 conn(0x7efc941430c0 legacy=0x7efc9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 10 monclient: _finish_hunting 0 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 1 monclient: found mon.noname-a 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940b44c0 con 0x7efc940a98f0 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 10 monclient: _finish_auth 0 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 10 monclient: _check_auth_tickets 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efc84001690 con 0x7efc940a98f0 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc99b90640 5 monclient: authenticate success, global_id 4284 2024-03-25T17:40:16.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc99b90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc927fc640 10 monclient: dump: 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:16.132 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:16.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.128+0000 7efc99b90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:16.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.145+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc840017f0 con 0x7efc940a98f0 2024-03-25T17:40:16.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.145+0000 7efc927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:16.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.145+0000 7efc927fc640 1 -- 172.21.15.44:0/2134173979 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efc84001c00 con 0x7efc940a98f0 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.145+0000 7efc927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.145+0000 7efc927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.145+0000 7efc927fc640 10 monclient: dump: 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:16.147 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:16.148 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.145+0000 7efc99b90640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.145+0000 7efc99b90640 4 set_mon_vals no callback set 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 10 monclient: shutdown 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 1 -- 172.21.15.44:0/2134173979 >> v1:172.21.15.44:6789/0 conn(0x7efc940a98f0 legacy=0x7efc940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 1 -- 172.21.15.44:0/2134173979 shutdown_connections 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 1 -- 172.21.15.44:0/2134173979 >> 172.21.15.44:0/2134173979 conn(0x7efc940b7ea0 msgr2=0x7efc940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 1 -- 172.21.15.44:0/2134173979 shutdown_connections 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 1 -- 172.21.15.44:0/2134173979 wait complete. 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 10 monclient: build_initial_monmap 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.146+0000 7efc99b90640 10 monclient: monmap: 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:16.150 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.147+0000 7efc99b90640 1 librados: starting msgr at 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.147+0000 7efc99b90640 1 librados: starting objecter 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.147+0000 7efc99b90640 1 Processor -- start 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.147+0000 7efc99b90640 1 -- start start 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.147+0000 7efc99b90640 1 librados: setting wanted keys 2024-03-25T17:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.147+0000 7efc99b90640 1 librados: calling monclient init 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.147+0000 7efc99b90640 10 monclient: init 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 10 monclient(hunting): picked mon.noname-b con 0x7efc940a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 10 monclient(hunting): picked mon.noname-a con 0x7efc9413f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 10 monclient(hunting): picked mon.noname-c con 0x7efc941430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc94135ac0 con 0x7efc9413f710 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc94137030 con 0x7efc940a98f0 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc941372a0 con 0x7efc941430c0 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.148+0000 7efc99b90640 10 monclient(hunting): authenticate will time out at 6019.376923s 2024-03-25T17:40:16.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.165+0000 7efc937fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efc940a98f0 0x7efc94133440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60332/0 (socket says 172.21.15.44:60332) 2024-03-25T17:40:16.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.165+0000 7efc937fe640 1 -- 172.21.15.44:0/3197429516 learned_addr learned my addr 172.21.15.44:0/3197429516 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:16.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.168+0000 7efc7bfff640 10 client.?.objecter ms_handle_connect 0x7efc940a98f0 2024-03-25T17:40:16.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.168+0000 7efc7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:16.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.175+0000 7efc7bfff640 10 client.?.objecter ms_handle_connect 0x7efc941430c0 2024-03-25T17:40:16.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.175+0000 7efc7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:16.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.175+0000 7efc7bfff640 10 client.?.objecter ms_handle_connect 0x7efc9413f710 2024-03-25T17:40:16.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.175+0000 7efc7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:16.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.182+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390419081 0 0) 0x7efc94137030 con 0x7efc940a98f0 2024-03-25T17:40:16.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.182+0000 7efc7bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:16.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.182+0000 7efc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:16.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.182+0000 7efc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:16.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.182+0000 7efc7bfff640 10 monclient(hunting): my global_id is 4273 2024-03-25T17:40:16.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.182+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc80004eb0 con 0x7efc940a98f0 2024-03-25T17:40:16.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.183+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437414859 0 0) 0x7efc941372a0 con 0x7efc941430c0 2024-03-25T17:40:16.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.183+0000 7efc7bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:16.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.183+0000 7efc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:16.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.183+0000 7efc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:16.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.183+0000 7efc7bfff640 10 monclient(hunting): my global_id is 4274 2024-03-25T17:40:16.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.183+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc94137030 con 0x7efc941430c0 2024-03-25T17:40:16.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.190+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698227293 0 0) 0x7efc94135ac0 con 0x7efc9413f710 2024-03-25T17:40:16.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.190+0000 7efc7bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:16.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.190+0000 7efc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:16.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.190+0000 7efc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:16.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.190+0000 7efc7bfff640 10 monclient(hunting): my global_id is 4287 2024-03-25T17:40:16.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.190+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc941372a0 con 0x7efc9413f710 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.194+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3184005153 0 0) 0x7efc80004eb0 con 0x7efc940a98f0 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.194+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc94135ac0 con 0x7efc940a98f0 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.194+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efc84001690 con 0x7efc940a98f0 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.194+0000 7efc7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.194+0000 7efc7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.194+0000 7efc7bfff640 10 monclient(hunting): dump: 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:16.197 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:16.197 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:16.197 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:16.197 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:16.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.194+0000 7efc7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:16.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.203+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1494482936 0 0) 0x7efc941372a0 con 0x7efc9413f710 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.203+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc80004eb0 con 0x7efc9413f710 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.203+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efc88001860 con 0x7efc9413f710 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.203+0000 7efc7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.203+0000 7efc7bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.203+0000 7efc7bfff640 10 monclient(hunting): dump: 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:16.205 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:16.206 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:16.206 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:16.206 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 160774942 0 0) 0x7efc94137030 con 0x7efc941430c0 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc941372a0 con 0x7efc941430c0 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efc8c001b00 con 0x7efc941430c0 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 10 monclient(hunting): dump: 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1667217744 0 0) 0x7efc94135ac0 con 0x7efc940a98f0 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 >> v1:172.21.15.44:6790/0 conn(0x7efc941430c0 legacy=0x7efc941e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 >> v1:172.21.15.44:6789/0 conn(0x7efc9413f710 legacy=0x7efc941379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 1 monclient: found mon.b 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc94137580 con 0x7efc940a98f0 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 10 monclient: _finish_auth 0 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 5 monclient: authenticate success, global_id 4273 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 10 monclient: _renew_subs 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc941363b0 con 0x7efc940a98f0 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 10 monclient: _renew_subs 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc941353a0 con 0x7efc940a98f0 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.205+0000 7efc99b90640 1 librados: init done 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.206+0000 7efc99b90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.206+0000 7efc99b90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.206+0000 7efc99b90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.206+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc94074cc0 con 0x7efc940a98f0 2024-03-25T17:40:16.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.418+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc840038a0 con 0x7efc940a98f0 2024-03-25T17:40:16.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.418+0000 7efc7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:16.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.418+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efc84003cf0 con 0x7efc940a98f0 2024-03-25T17:40:16.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.418+0000 7efc7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.418+0000 7efc7bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.418+0000 7efc7bfff640 10 monclient: dump: 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:16.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.418+0000 7efc91ffb640 4 set_mon_vals no callback set 2024-03-25T17:40:16.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.483+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7efc84003e50 con 0x7efc940a98f0 2024-03-25T17:40:16.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.483+0000 7efc92ffd640 10 monclient: get_auth_request con 0x7efc8006e0f0 auth_method 0 2024-03-25T17:40:16.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.483+0000 7efc7bfff640 10 client.4273.objecter ms_handle_connect 0x7efc8006e0f0 2024-03-25T17:40:16.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.496+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7efc84054970 con 0x7efc940a98f0 2024-03-25T17:40:16.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.496+0000 7efc7bfff640 10 client.4273.objecter ms_dispatch 0x7efc94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:40:16.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.496+0000 7efc7bfff640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:40:16.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.496+0000 7efc7bfff640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:40:16.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.496+0000 7efc7bfff640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-25T17:40:16.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.498+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc84050480 con 0x7efc940a98f0 2024-03-25T17:40:16.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.498+0000 7efc7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:16.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.498+0000 7efc7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:16.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.743+0000 7efc99b90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:16.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.743+0000 7efc99b90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:16.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.743+0000 7efc99b90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:16.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.743+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc940627b0 con 0x7efc940a98f0 2024-03-25T17:40:16.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.769+0000 7efc7bfff640 1 -- 172.21.15.44:0/3197429516 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7efc840523b0 con 0x7efc940a98f0 2024-03-25T17:40:16.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.769+0000 7efc7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:16.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.769+0000 7efc7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:16.771 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:40:16.771 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:16.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 10 librados: watch_flush enter 2024-03-25T17:40:16.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 10 librados: watch_flush exit 2024-03-25T17:40:16.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-25T17:40:16.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 10 client.4273.objecter successfully canceled tick 2024-03-25T17:40:16.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 >> v1:172.21.15.44:6800/3257043132 conn(0x7efc8006e0f0 legacy=0x7efc800705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 10 monclient: shutdown 2024-03-25T17:40:16.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:16.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 >> v1:172.21.15.111:6789/0 conn(0x7efc940a98f0 legacy=0x7efc94133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 shutdown_connections 2024-03-25T17:40:16.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 >> 172.21.15.44:0/3197429516 conn(0x7efc940b7ea0 msgr2=0x7efc94074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:16.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 shutdown_connections 2024-03-25T17:40:16.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 1 -- 172.21.15.44:0/3197429516 wait complete. 2024-03-25T17:40:16.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:16.772+0000 7efc99b90640 1 librados: shutdown 2024-03-25T17:40:16.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:17.290 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:34.646 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:17.298 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:34.653 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:17.344 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:34.699 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:17.347 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:34.703 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:17.350 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:34.705 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:17.361 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:34.681 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:17.368 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:34.688 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:17.410 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:34.766 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:17.427 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:34.747 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:23.594 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:23.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.711+0000 7f3947b7b640 10 monclient: get_monmap_and_config 2024-03-25T17:40:23.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.711+0000 7f3947b7b640 10 monclient: build_initial_monmap 2024-03-25T17:40:23.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.711+0000 7f3947b7b640 10 monclient: monmap: 2024-03-25T17:40:23.712 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:23.712 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:40:23.712061+0000 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:40:23.712061+0000 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:23.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.711+0000 7f3947b7b640 1 Processor -- start 2024-03-25T17:40:23.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 1 -- start start 2024-03-25T17:40:23.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 10 monclient: init 2024-03-25T17:40:23.715 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:23.715 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:40:23.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 10 monclient(hunting): picked mon.noname-c con 0x7f39400a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:23.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 10 monclient(hunting): picked mon.noname-a con 0x7f394013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:23.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 10 monclient(hunting): picked mon.noname-b con 0x7f39401430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:23.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3940142520 con 0x7f394013f710 2024-03-25T17:40:23.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b28a0 con 0x7f39400a98f0 2024-03-25T17:40:23.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b2a30 con 0x7f39401430c0 2024-03-25T17:40:23.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:23.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.712+0000 7f3947b7b640 10 monclient(hunting): authenticate will time out at 6026.940607s 2024-03-25T17:40:23.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.719+0000 7f3945917640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f39400a98f0 0x7f39400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:53944/0 (socket says 172.21.15.44:53944) 2024-03-25T17:40:23.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.719+0000 7f3945917640 1 -- 172.21.15.44:0/3071319231 learned_addr learned my addr 172.21.15.44:0/3071319231 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:23.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.730+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800985993 0 0) 0x7f39400b28a0 con 0x7f39400a98f0 2024-03-25T17:40:23.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.730+0000 7f3944915640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.730+0000 7f3944915640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.730+0000 7f3944915640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.730+0000 7f3944915640 10 monclient(hunting): my global_id is 4276 2024-03-25T17:40:23.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.730+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f392c003520 con 0x7f39400a98f0 2024-03-25T17:40:23.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.733+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995174759 0 0) 0x7f39400b2a30 con 0x7f39401430c0 2024-03-25T17:40:23.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.733+0000 7f3944915640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.733+0000 7f3944915640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.733+0000 7f3944915640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.733+0000 7f3944915640 10 monclient(hunting): my global_id is 4277 2024-03-25T17:40:23.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.733+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39400b28a0 con 0x7f39401430c0 2024-03-25T17:40:23.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.735+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198984390 0 0) 0x7f3940142520 con 0x7f394013f710 2024-03-25T17:40:23.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.735+0000 7f3944915640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.735+0000 7f3944915640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.735+0000 7f3944915640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.735+0000 7f3944915640 10 monclient(hunting): my global_id is 4290 2024-03-25T17:40:23.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.735+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39400b2a30 con 0x7f394013f710 2024-03-25T17:40:23.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1945428145 0 0) 0x7f392c003520 con 0x7f39400a98f0 2024-03-25T17:40:23.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 >> v1:172.21.15.44:6790/0 conn(0x7f39401430c0 legacy=0x7f394014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 >> v1:172.21.15.44:6789/0 conn(0x7f394013f710 legacy=0x7f39401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 10 monclient: _finish_hunting 0 2024-03-25T17:40:23.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 1 monclient: found mon.noname-c 2024-03-25T17:40:23.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:23.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:40:23.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39400b44c0 con 0x7f39400a98f0 2024-03-25T17:40:23.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 10 monclient: _finish_auth 0 2024-03-25T17:40:23.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 10 monclient: _check_auth_tickets 2024-03-25T17:40:23.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:23.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f393c001690 con 0x7f39400a98f0 2024-03-25T17:40:23.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3947b7b640 5 monclient: authenticate success, global_id 4276 2024-03-25T17:40:23.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3947b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3944915640 10 monclient: dump: 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:23.743 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:23.744 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:23.744 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:23.744 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:23.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.740+0000 7f3947b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:23.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.754+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f393c0017f0 con 0x7f39400a98f0 2024-03-25T17:40:23.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.754+0000 7f3944915640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:23.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.754+0000 7f3944915640 1 -- 172.21.15.44:0/3071319231 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f393c001c00 con 0x7f39400a98f0 2024-03-25T17:40:23.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.754+0000 7f3947b7b640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:23.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.754+0000 7f3947b7b640 4 set_mon_vals no callback set 2024-03-25T17:40:23.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.754+0000 7f3944915640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:23.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.754+0000 7f3944915640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:23.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.754+0000 7f3944915640 10 monclient: dump: 2024-03-25T17:40:23.755 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:23.756 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:23.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.755+0000 7f3947b7b640 10 monclient: shutdown 2024-03-25T17:40:23.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.755+0000 7f3947b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:23.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.755+0000 7f3947b7b640 1 -- 172.21.15.44:0/3071319231 >> v1:172.21.15.111:6789/0 conn(0x7f39400a98f0 legacy=0x7f39400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.755+0000 7f3947b7b640 1 -- 172.21.15.44:0/3071319231 shutdown_connections 2024-03-25T17:40:23.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.755+0000 7f3947b7b640 1 -- 172.21.15.44:0/3071319231 >> 172.21.15.44:0/3071319231 conn(0x7f39400b7ea0 msgr2=0x7f39400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:23.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.755+0000 7f3947b7b640 1 -- 172.21.15.44:0/3071319231 shutdown_connections 2024-03-25T17:40:23.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.755+0000 7f3947b7b640 1 -- 172.21.15.44:0/3071319231 wait complete. 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 10 monclient: build_initial_monmap 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 10 monclient: monmap: 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:40:23.761 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:23.762 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:40:23.762 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:40:23.762 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:23.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 1 librados: starting msgr at 2024-03-25T17:40:23.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 1 librados: starting objecter 2024-03-25T17:40:23.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 1 Processor -- start 2024-03-25T17:40:23.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 1 -- start start 2024-03-25T17:40:23.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 1 librados: setting wanted keys 2024-03-25T17:40:23.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 1 librados: calling monclient init 2024-03-25T17:40:23.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.756+0000 7f3947b7b640 10 monclient: init 2024-03-25T17:40:23.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:23.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:40:23.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 10 monclient(hunting): picked mon.noname-c con 0x7f39400a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:23.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 10 monclient(hunting): picked mon.noname-a con 0x7f394013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:23.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 10 monclient(hunting): picked mon.noname-b con 0x7f39401430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:23.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39401d7380 con 0x7f394013f710 2024-03-25T17:40:23.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3940139840 con 0x7f39401430c0 2024-03-25T17:40:23.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39401d8200 con 0x7f39400a98f0 2024-03-25T17:40:23.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:23.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.757+0000 7f3947b7b640 10 monclient(hunting): authenticate will time out at 6026.985044s 2024-03-25T17:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.762+0000 7f3946118640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f39401430c0 0x7f39401daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:53984/0 (socket says 172.21.15.44:53984) 2024-03-25T17:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.762+0000 7f3946118640 1 -- 172.21.15.44:0/1748737662 learned_addr learned my addr 172.21.15.44:0/1748737662 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.763+0000 7f39367fc640 10 client.?.objecter ms_handle_connect 0x7f39401430c0 2024-03-25T17:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.763+0000 7f39367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:23.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.771+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793381752 0 0) 0x7f3940139840 con 0x7f39401430c0 2024-03-25T17:40:23.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.771+0000 7f39367fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.771+0000 7f39367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.771+0000 7f39367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.771+0000 7f39367fc640 10 monclient(hunting): my global_id is 4279 2024-03-25T17:40:23.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.771+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3924004eb0 con 0x7f39401430c0 2024-03-25T17:40:23.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.772+0000 7f39367fc640 10 client.?.objecter ms_handle_connect 0x7f39400a98f0 2024-03-25T17:40:23.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.772+0000 7f39367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:23.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.773+0000 7f39367fc640 10 client.?.objecter ms_handle_connect 0x7f394013f710 2024-03-25T17:40:23.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.773+0000 7f39367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:23.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.785+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978594632 0 0) 0x7f3924004eb0 con 0x7f39401430c0 2024-03-25T17:40:23.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.785+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3940139840 con 0x7f39401430c0 2024-03-25T17:40:23.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.785+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f3938001c90 con 0x7f39401430c0 2024-03-25T17:40:23.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.785+0000 7f39367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:23.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.785+0000 7f39367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:23.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.785+0000 7f39367fc640 10 monclient(hunting): dump: 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:23.788 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:23.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.785+0000 7f39367fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:23.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.793+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128064947 0 0) 0x7f39401d8200 con 0x7f39400a98f0 2024-03-25T17:40:23.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.793+0000 7f39367fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.793+0000 7f39367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.793+0000 7f39367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.793+0000 7f39367fc640 10 monclient(hunting): my global_id is 4280 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.793+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3924004eb0 con 0x7f39400a98f0 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.794+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871202329 0 0) 0x7f39401d7380 con 0x7f394013f710 2024-03-25T17:40:23.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.794+0000 7f39367fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.794+0000 7f39367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.794+0000 7f39367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.794+0000 7f39367fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T17:40:23.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.794+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39401d8200 con 0x7f394013f710 2024-03-25T17:40:23.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3308900214 0 0) 0x7f3940139840 con 0x7f39401430c0 2024-03-25T17:40:23.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 >> v1:172.21.15.44:6790/0 conn(0x7f39400a98f0 legacy=0x7f39401d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 >> v1:172.21.15.44:6789/0 conn(0x7f394013f710 legacy=0x7f39401da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 10 monclient: _finish_hunting 0 2024-03-25T17:40:23.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 1 monclient: found mon.b 2024-03-25T17:40:23.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:23.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:23.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39401d75d0 con 0x7f39401430c0 2024-03-25T17:40:23.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 10 monclient: _finish_auth 0 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 10 monclient: _check_auth_tickets 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f39367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f3947b7b640 5 monclient: authenticate success, global_id 4279 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f3947b7b640 10 monclient: _renew_subs 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f3947b7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.801+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39401db6a0 con 0x7f39401430c0 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 10 monclient: _renew_subs 2024-03-25T17:40:23.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:23.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3940137910 con 0x7f39401430c0 2024-03-25T17:40:23.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 1 librados: init done 2024-03-25T17:40:23.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:23.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:23.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:23.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:23.802+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39400627b0 con 0x7f39401430c0 2024-03-25T17:40:24.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.072+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3938003970 con 0x7f39401430c0 2024-03-25T17:40:24.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.072+0000 7f39367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.072+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f3938003d00 con 0x7f39401430c0 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.072+0000 7f39367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.072+0000 7f39367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.072+0000 7f39367fc640 10 monclient: dump: 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:24.073 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:24.074 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:24.074 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:24.074 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:24.074 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:24.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.072+0000 7f3937fff640 4 set_mon_vals no callback set 2024-03-25T17:40:24.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.073+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f3938003f20 con 0x7f39401430c0 2024-03-25T17:40:24.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.073+0000 7f3945917640 10 monclient: get_auth_request con 0x7f392406de80 auth_method 0 2024-03-25T17:40:24.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.074+0000 7f39367fc640 10 client.4279.objecter ms_handle_connect 0x7f392406de80 2024-03-25T17:40:24.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.082+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f3938054ad0 con 0x7f39401430c0 2024-03-25T17:40:24.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.082+0000 7f39367fc640 10 client.4279.objecter ms_dispatch 0x7f3940000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:40:24.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.082+0000 7f39367fc640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:40:24.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.082+0000 7f39367fc640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:40:24.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.082+0000 7f39367fc640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-25T17:40:24.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.083+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39380505e0 con 0x7f39401430c0 2024-03-25T17:40:24.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.083+0000 7f39367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:24.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.083+0000 7f39367fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:24.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.315+0000 7f3947b7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:24.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.315+0000 7f3947b7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:24.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.315+0000 7f3947b7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:24.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.315+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3940069f40 con 0x7f39401430c0 2024-03-25T17:40:24.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.472+0000 7f39367fc640 1 -- 172.21.15.44:0/1748737662 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f3938052510 con 0x7f39401430c0 2024-03-25T17:40:24.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.472+0000 7f39367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:24.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.472+0000 7f39367fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:24.473 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:40:24.473 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:24.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 10 librados: watch_flush enter 2024-03-25T17:40:24.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 10 librados: watch_flush exit 2024-03-25T17:40:24.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 10 client.4279.objecter successfully canceled tick 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 >> v1:172.21.15.44:6800/3257043132 conn(0x7f392406de80 legacy=0x7f3924070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 10 monclient: shutdown 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 >> v1:172.21.15.111:6789/0 conn(0x7f39401430c0 legacy=0x7f39401daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 shutdown_connections 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 >> 172.21.15.44:0/1748737662 conn(0x7f39400b7ea0 msgr2=0x7f3940074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 shutdown_connections 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 1 -- 172.21.15.44:0/1748737662 wait complete. 2024-03-25T17:40:24.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:24.474+0000 7f3947b7b640 1 librados: shutdown 2024-03-25T17:40:24.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:31.293 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.408+0000 7fcb101d6640 10 monclient: get_monmap_and_config 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.408+0000 7fcb101d6640 10 monclient: build_initial_monmap 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.408+0000 7fcb101d6640 10 monclient: monmap: 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:40:31.409218+0000 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:40:31.409218+0000 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:31.409 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:40:31.410 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:40:31.410 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:31.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.408+0000 7fcb101d6640 1 Processor -- start 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.409+0000 7fcb101d6640 1 -- start start 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.409+0000 7fcb101d6640 10 monclient: init 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.409+0000 7fcb101d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.409+0000 7fcb101d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.409+0000 7fcb101d6640 10 monclient(hunting): picked mon.noname-a con 0x7fcb080a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.409+0000 7fcb101d6640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.410+0000 7fcb101d6640 10 monclient(hunting): picked mon.noname-c con 0x7fcb081430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.410+0000 7fcb101d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb08142520 con 0x7fcb080a98f0 2024-03-25T17:40:31.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.410+0000 7fcb101d6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb080b28a0 con 0x7fcb081430c0 2024-03-25T17:40:31.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.410+0000 7fcb101d6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb080b2a30 con 0x7fcb0813f710 2024-03-25T17:40:31.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.410+0000 7fcb101d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:31.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.410+0000 7fcb101d6640 10 monclient(hunting): authenticate will time out at 6034.637767s 2024-03-25T17:40:31.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.426+0000 7fcb0e773640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcb081430c0 0x7fcb0814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46888/0 (socket says 172.21.15.44:46888) 2024-03-25T17:40:31.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.426+0000 7fcb0e773640 1 -- 172.21.15.44:0/1265635020 learned_addr learned my addr 172.21.15.44:0/1265635020 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:31.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.433+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935623354 0 0) 0x7fcb080b2a30 con 0x7fcb0813f710 2024-03-25T17:40:31.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.433+0000 7fcb0cf70640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.433+0000 7fcb0cf70640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.433+0000 7fcb0cf70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.433+0000 7fcb0cf70640 10 monclient(hunting): my global_id is 4283 2024-03-25T17:40:31.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.433+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf4003520 con 0x7fcb0813f710 2024-03-25T17:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.446+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181990372 0 0) 0x7fcb080b28a0 con 0x7fcb081430c0 2024-03-25T17:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.446+0000 7fcb0cf70640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.446+0000 7fcb0cf70640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.446+0000 7fcb0cf70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.446+0000 7fcb0cf70640 10 monclient(hunting): my global_id is 4282 2024-03-25T17:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.446+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb080b2a30 con 0x7fcb081430c0 2024-03-25T17:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3575718436 0 0) 0x7fcaf4003520 con 0x7fcb0813f710 2024-03-25T17:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 >> v1:172.21.15.111:6789/0 conn(0x7fcb081430c0 legacy=0x7fcb0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 >> v1:172.21.15.44:6789/0 conn(0x7fcb080a98f0 legacy=0x7fcb080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 10 monclient: _finish_hunting 0 2024-03-25T17:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 1 monclient: found mon.noname-b 2024-03-25T17:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb080b44c0 con 0x7fcb0813f710 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 10 monclient: _finish_auth 0 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 10 monclient: _check_auth_tickets 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fcaf8001690 con 0x7fcb0813f710 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb0cf70640 10 monclient: dump: 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:31.449 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:31.450 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:31.450 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:31.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb101d6640 5 monclient: authenticate success, global_id 4283 2024-03-25T17:40:31.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.447+0000 7fcb101d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:31.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.458+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaf80017f0 con 0x7fcb0813f710 2024-03-25T17:40:31.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.458+0000 7fcb0cf70640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.458+0000 7fcb0cf70640 1 -- 172.21.15.44:0/1265635020 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fcaf8001c00 con 0x7fcb0813f710 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.458+0000 7fcb101d6640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.458+0000 7fcb101d6640 4 set_mon_vals no callback set 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.458+0000 7fcb0cf70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.458+0000 7fcb0cf70640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.458+0000 7fcb0cf70640 10 monclient: dump: 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:31.459 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:31.460 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:31.460 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:31.460 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.459+0000 7fcb101d6640 10 monclient: shutdown 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.459+0000 7fcb101d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.459+0000 7fcb101d6640 1 -- 172.21.15.44:0/1265635020 >> v1:172.21.15.44:6790/0 conn(0x7fcb0813f710 legacy=0x7fcb081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.459+0000 7fcb101d6640 1 -- 172.21.15.44:0/1265635020 shutdown_connections 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.459+0000 7fcb101d6640 1 -- 172.21.15.44:0/1265635020 >> 172.21.15.44:0/1265635020 conn(0x7fcb080b7ea0 msgr2=0x7fcb080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.459+0000 7fcb101d6640 1 -- 172.21.15.44:0/1265635020 shutdown_connections 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.459+0000 7fcb101d6640 1 -- 172.21.15.44:0/1265635020 wait complete. 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.460+0000 7fcb101d6640 10 monclient: build_initial_monmap 2024-03-25T17:40:31.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.460+0000 7fcb101d6640 10 monclient: monmap: 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.460+0000 7fcb101d6640 1 librados: starting msgr at 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.460+0000 7fcb101d6640 1 librados: starting objecter 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.461+0000 7fcb101d6640 1 Processor -- start 2024-03-25T17:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.461+0000 7fcb101d6640 1 -- start start 2024-03-25T17:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.461+0000 7fcb101d6640 1 librados: setting wanted keys 2024-03-25T17:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.461+0000 7fcb101d6640 1 librados: calling monclient init 2024-03-25T17:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.461+0000 7fcb101d6640 10 monclient: init 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 10 monclient(hunting): picked mon.noname-a con 0x7fcb080a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 10 monclient(hunting): picked mon.noname-c con 0x7fcb081430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb08137df0 con 0x7fcb080a98f0 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb081337a0 con 0x7fcb0813f710 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb08133930 con 0x7fcb081430c0 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.462+0000 7fcb101d6640 10 monclient(hunting): authenticate will time out at 6034.690059s 2024-03-25T17:40:31.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.476+0000 7fcb0df72640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fcb080a98f0 0x7fcb081392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45276/0 (socket says 172.21.15.44:45276) 2024-03-25T17:40:31.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.476+0000 7fcb0df72640 1 -- 172.21.15.44:0/3247777186 learned_addr learned my addr 172.21.15.44:0/3247777186 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:31.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.477+0000 7fcafe7fc640 10 client.?.objecter ms_handle_connect 0x7fcb080a98f0 2024-03-25T17:40:31.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.477+0000 7fcafe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.485+0000 7fcafe7fc640 10 client.?.objecter ms_handle_connect 0x7fcb0813f710 2024-03-25T17:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.485+0000 7fcafe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.487+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024407279 0 0) 0x7fcb08137df0 con 0x7fcb080a98f0 2024-03-25T17:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.487+0000 7fcafe7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.487+0000 7fcafe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.487+0000 7fcafe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.487+0000 7fcafe7fc640 10 monclient(hunting): my global_id is 4299 2024-03-25T17:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.487+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaec004eb0 con 0x7fcb080a98f0 2024-03-25T17:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.501+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381047082 0 0) 0x7fcb081337a0 con 0x7fcb0813f710 2024-03-25T17:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.501+0000 7fcafe7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.501+0000 7fcafe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.501+0000 7fcafe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.501+0000 7fcafe7fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T17:40:31.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.501+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb08137df0 con 0x7fcb0813f710 2024-03-25T17:40:31.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.501+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 591127949 0 0) 0x7fcaec004eb0 con 0x7fcb080a98f0 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.502+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb081337a0 con 0x7fcb080a98f0 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.502+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fcb04001db0 con 0x7fcb080a98f0 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.502+0000 7fcafe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.502+0000 7fcafe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.502+0000 7fcafe7fc640 10 monclient(hunting): dump: 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:31.504 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:31.504 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:31.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.502+0000 7fcafe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:31.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.503+0000 7fcafe7fc640 10 client.?.objecter ms_handle_connect 0x7fcb081430c0 2024-03-25T17:40:31.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.503+0000 7fcafe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:31.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.507+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742408669 0 0) 0x7fcb08133930 con 0x7fcb081430c0 2024-03-25T17:40:31.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.507+0000 7fcafe7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.507+0000 7fcafe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.507+0000 7fcafe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.507+0000 7fcafe7fc640 10 monclient(hunting): my global_id is 4286 2024-03-25T17:40:31.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.507+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaec004eb0 con 0x7fcb081430c0 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1857895711 0 0) 0x7fcb081337a0 con 0x7fcb080a98f0 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 >> v1:172.21.15.44:6790/0 conn(0x7fcb081430c0 legacy=0x7fcb08133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 >> v1:172.21.15.111:6789/0 conn(0x7fcb0813f710 legacy=0x7fcb081376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 1 monclient: found mon.a 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb081367d0 con 0x7fcb080a98f0 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 10 monclient: _finish_auth 0 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcafe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 5 monclient: authenticate success, global_id 4299 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 10 monclient: _renew_subs 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb081369e0 con 0x7fcb080a98f0 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 10 monclient: _renew_subs 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0813b640 con 0x7fcb080a98f0 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.518+0000 7fcb101d6640 1 librados: init done 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.519+0000 7fcb101d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.519+0000 7fcb101d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.519+0000 7fcb101d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.519+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb080627b0 con 0x7fcb080a98f0 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.835+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb04001ff0 con 0x7fcb080a98f0 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.835+0000 7fcafe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.835+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fcb04003de0 con 0x7fcb080a98f0 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.835+0000 7fcafe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.835+0000 7fcafe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.835+0000 7fcafe7fc640 10 monclient: dump: 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:31.836 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:31.837 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:31.837 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:31.837 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:31.837 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:31.837 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:31.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.835+0000 7fcafffff640 4 set_mon_vals no callback set 2024-03-25T17:40:31.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.836+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fcb04003fc0 con 0x7fcb080a98f0 2024-03-25T17:40:31.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.837+0000 7fcb0d771640 10 monclient: get_auth_request con 0x7fcaec06ddf0 auth_method 0 2024-03-25T17:40:31.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.837+0000 7fcafe7fc640 10 client.4299.objecter ms_handle_connect 0x7fcaec06ddf0 2024-03-25T17:40:31.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.843+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fcb04054c30 con 0x7fcb080a98f0 2024-03-25T17:40:31.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.843+0000 7fcafe7fc640 10 client.4299.objecter ms_dispatch 0x7fcb08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:40:31.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.843+0000 7fcafe7fc640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:40:31.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.843+0000 7fcafe7fc640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:40:31.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.843+0000 7fcafe7fc640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-25T17:40:31.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.843+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb04050740 con 0x7fcb080a98f0 2024-03-25T17:40:31.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.843+0000 7fcafe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:31.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:31.843+0000 7fcafe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.076+0000 7fcb101d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.076+0000 7fcb101d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.076+0000 7fcb101d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.076+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb08137300 con 0x7fcb080a98f0 2024-03-25T17:40:32.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.147+0000 7fcafe7fc640 1 -- 172.21.15.44:0/3247777186 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fcb04052670 con 0x7fcb080a98f0 2024-03-25T17:40:32.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.147+0000 7fcafe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:32.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.147+0000 7fcafe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:32.148 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:40:32.148 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:32.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 10 librados: watch_flush enter 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 10 librados: watch_flush exit 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 10 client.4299.objecter successfully canceled tick 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 >> v1:172.21.15.44:6800/3257043132 conn(0x7fcaec06ddf0 legacy=0x7fcaec0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 10 monclient: shutdown 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 >> v1:172.21.15.44:6789/0 conn(0x7fcb080a98f0 legacy=0x7fcb081392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 shutdown_connections 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.149+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 >> 172.21.15.44:0/3247777186 conn(0x7fcb080b7ea0 msgr2=0x7fcb08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.150+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 shutdown_connections 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.150+0000 7fcb101d6640 1 -- 172.21.15.44:0/3247777186 wait complete. 2024-03-25T17:40:32.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:32.150+0000 7fcb101d6640 1 librados: shutdown 2024-03-25T17:40:32.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:37.253 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:54.609 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:37.258 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:54.614 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:37.296 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:54.651 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:37.330 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:54.686 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:37.338 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:54.694 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:37.340 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:54.660 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:37.348 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:54.668 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:37.381 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:54.735 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:37.382 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:54.701 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:38.969 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:39.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.087+0000 7fcec2e74640 10 monclient: get_monmap_and_config 2024-03-25T17:40:39.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.087+0000 7fcec2e74640 10 monclient: build_initial_monmap 2024-03-25T17:40:39.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.087+0000 7fcec2e74640 10 monclient: monmap: 2024-03-25T17:40:39.087 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:39.087 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:39.087 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:40:39.087962+0000 2024-03-25T17:40:39.087 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:40:39.087962+0000 2024-03-25T17:40:39.088 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:39.088 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:39.088 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:39.088 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:40:39.088 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:40:39.088 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.087+0000 7fcec2e74640 1 Processor -- start 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.087+0000 7fcec2e74640 1 -- start start 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.087+0000 7fcec2e74640 10 monclient: init 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 10 monclient(hunting): picked mon.noname-a con 0x7fcebc0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 10 monclient(hunting): picked mon.noname-c con 0x7fcebc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 10 monclient(hunting): picked mon.noname-b con 0x7fcebc1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:39.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc142520 con 0x7fcebc0a98f0 2024-03-25T17:40:39.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0b28a0 con 0x7fcebc13f710 2024-03-25T17:40:39.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0b2a30 con 0x7fcebc1430c0 2024-03-25T17:40:39.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:39.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.088+0000 7fcec2e74640 10 monclient(hunting): authenticate will time out at 6042.316309s 2024-03-25T17:40:39.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.095+0000 7fcec1411640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fcebc1430c0 0x7fcebc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:34284/0 (socket says 172.21.15.44:34284) 2024-03-25T17:40:39.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.095+0000 7fcec1411640 1 -- 172.21.15.44:0/3245311412 learned_addr learned my addr 172.21.15.44:0/3245311412 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:39.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.104+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944430841 0 0) 0x7fcebc0b28a0 con 0x7fcebc13f710 2024-03-25T17:40:39.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.104+0000 7fcebb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:40:39.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.104+0000 7fcebb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:39.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.104+0000 7fcebb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:39.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.104+0000 7fcebb7fe640 10 monclient(hunting): my global_id is 4288 2024-03-25T17:40:39.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.104+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceac003520 con 0x7fcebc13f710 2024-03-25T17:40:39.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.107+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195220018 0 0) 0x7fcebc0b2a30 con 0x7fcebc1430c0 2024-03-25T17:40:39.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.107+0000 7fcebb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:40:39.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.107+0000 7fcebb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:39.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.107+0000 7fcebb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:39.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.107+0000 7fcebb7fe640 10 monclient(hunting): my global_id is 4289 2024-03-25T17:40:39.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.107+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcebc0b28a0 con 0x7fcebc1430c0 2024-03-25T17:40:39.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 933267304 0 0) 0x7fcebc0b28a0 con 0x7fcebc1430c0 2024-03-25T17:40:39.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 >> v1:172.21.15.111:6789/0 conn(0x7fcebc13f710 legacy=0x7fcebc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:39.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 >> v1:172.21.15.44:6789/0 conn(0x7fcebc0a98f0 legacy=0x7fcebc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:39.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:40:39.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 1 monclient: found mon.noname-b 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc0b44c0 con 0x7fcebc1430c0 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 10 monclient: _finish_auth 0 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fceb40019f0 con 0x7fcebc1430c0 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcec2e74640 5 monclient: authenticate success, global_id 4289 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcec2e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcebb7fe640 10 monclient: dump: 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:39.117 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.118 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.118 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:39.118 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:39.118 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:39.118 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:39.118 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:39.118 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:39.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.116+0000 7fcec2e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:39.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.125+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fceb4001b50 con 0x7fcebc1430c0 2024-03-25T17:40:39.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.125+0000 7fcebb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:39.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.125+0000 7fcebb7fe640 1 -- 172.21.15.44:0/3245311412 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fceb4001f20 con 0x7fcebc1430c0 2024-03-25T17:40:39.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.125+0000 7fcec2e74640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.125+0000 7fcec2e74640 4 set_mon_vals no callback set 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.125+0000 7fcebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.125+0000 7fcebb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.125+0000 7fcebb7fe640 10 monclient: dump: 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:39.126 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:39.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.126+0000 7fcec2e74640 10 monclient: shutdown 2024-03-25T17:40:39.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.126+0000 7fcec2e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:39.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.126+0000 7fcec2e74640 1 -- 172.21.15.44:0/3245311412 >> v1:172.21.15.44:6790/0 conn(0x7fcebc1430c0 legacy=0x7fcebc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.126+0000 7fcec2e74640 1 -- 172.21.15.44:0/3245311412 shutdown_connections 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.126+0000 7fcec2e74640 1 -- 172.21.15.44:0/3245311412 >> 172.21.15.44:0/3245311412 conn(0x7fcebc0b7ea0 msgr2=0x7fcebc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.126+0000 7fcec2e74640 1 -- 172.21.15.44:0/3245311412 shutdown_connections 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.126+0000 7fcec2e74640 1 -- 172.21.15.44:0/3245311412 wait complete. 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.126+0000 7fcec2e74640 10 monclient: build_initial_monmap 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.127+0000 7fcec2e74640 10 monclient: monmap: 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:39.129 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.127+0000 7fcec2e74640 1 librados: starting msgr at 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.127+0000 7fcec2e74640 1 librados: starting objecter 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.127+0000 7fcec2e74640 1 Processor -- start 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.127+0000 7fcec2e74640 1 -- start start 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.127+0000 7fcec2e74640 1 librados: setting wanted keys 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.127+0000 7fcec2e74640 1 librados: calling monclient init 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.127+0000 7fcec2e74640 10 monclient: init 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 10 monclient(hunting): picked mon.noname-a con 0x7fcebc0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 10 monclient(hunting): picked mon.noname-c con 0x7fcebc13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:39.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 10 monclient(hunting): picked mon.noname-b con 0x7fcebc1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:39.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc137e20 con 0x7fcebc0a98f0 2024-03-25T17:40:39.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0b1720 con 0x7fcebc1430c0 2024-03-25T17:40:39.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc138710 con 0x7fcebc13f710 2024-03-25T17:40:39.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:39.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.128+0000 7fcec2e74640 10 monclient(hunting): authenticate will time out at 6042.356255s 2024-03-25T17:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.141+0000 7fcec1411640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcebc1430c0 0x7fcebc1daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:51960/0 (socket says 172.21.15.44:51960) 2024-03-25T17:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.141+0000 7fcec1411640 1 -- 172.21.15.44:0/139708705 learned_addr learned my addr 172.21.15.44:0/139708705 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.141+0000 7fceb97fa640 10 client.?.objecter ms_handle_connect 0x7fcebc1430c0 2024-03-25T17:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.141+0000 7fceb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:39.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.154+0000 7fceb97fa640 10 client.?.objecter ms_handle_connect 0x7fcebc13f710 2024-03-25T17:40:39.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.154+0000 7fceb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:39.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.160+0000 7fceb97fa640 10 client.?.objecter ms_handle_connect 0x7fcebc0a98f0 2024-03-25T17:40:39.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.160+0000 7fceb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:39.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.161+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753407046 0 0) 0x7fcebc0b1720 con 0x7fcebc1430c0 2024-03-25T17:40:39.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.161+0000 7fceb97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:40:39.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.161+0000 7fceb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:39.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.161+0000 7fceb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:39.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.161+0000 7fceb97fa640 10 monclient(hunting): my global_id is 4291 2024-03-25T17:40:39.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.161+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea0004eb0 con 0x7fcebc1430c0 2024-03-25T17:40:39.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.174+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057807838 0 0) 0x7fcebc138710 con 0x7fcebc13f710 2024-03-25T17:40:39.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.174+0000 7fceb97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:40:39.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.174+0000 7fceb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:39.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.174+0000 7fceb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:39.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.174+0000 7fceb97fa640 10 monclient(hunting): my global_id is 4292 2024-03-25T17:40:39.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.174+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcebc0b1720 con 0x7fcebc13f710 2024-03-25T17:40:39.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.176+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216151394 0 0) 0x7fcebc137e20 con 0x7fcebc0a98f0 2024-03-25T17:40:39.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.176+0000 7fceb97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:40:39.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.176+0000 7fceb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:39.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.176+0000 7fceb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:39.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.176+0000 7fceb97fa640 10 monclient(hunting): my global_id is 4305 2024-03-25T17:40:39.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.176+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcebc138710 con 0x7fcebc0a98f0 2024-03-25T17:40:39.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.190+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2860507156 0 0) 0x7fcebc0b1720 con 0x7fcebc13f710 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.190+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcebc137e20 con 0x7fcebc13f710 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.190+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fcea8001950 con 0x7fcebc13f710 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.190+0000 7fceb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.190+0000 7fceb97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.190+0000 7fceb97fa640 10 monclient(hunting): dump: 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:39.191 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:39.192 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:39.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.190+0000 7fceb97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:39.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.194+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262134336 0 0) 0x7fcea0004eb0 con 0x7fcebc1430c0 2024-03-25T17:40:39.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.194+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcebc0b1720 con 0x7fcebc1430c0 2024-03-25T17:40:39.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.194+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fceb4001f20 con 0x7fcebc1430c0 2024-03-25T17:40:39.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.194+0000 7fceb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:39.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.194+0000 7fceb97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:39.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.194+0000 7fceb97fa640 10 monclient(hunting): dump: 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:39.195 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.201+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1244521620 0 0) 0x7fcebc138710 con 0x7fcebc0a98f0 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.201+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcea0004eb0 con 0x7fcebc0a98f0 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.201+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fceb0001870 con 0x7fcebc0a98f0 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.201+0000 7fceb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.201+0000 7fceb97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.201+0000 7fceb97fa640 10 monclient(hunting): dump: 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.202 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:39.203 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:39.203 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:39.203 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:39.203 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:39.203 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.214+0000 7fcec0c10640 1 -- 172.21.15.44:0/139708705 >> v1:172.21.15.44:6789/0 conn(0x7fcebc0a98f0 legacy=0x7fcebc137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T17:40:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.214+0000 7fcec0c10640 1 -- 172.21.15.44:0/139708705 >> v1:172.21.15.44:6789/0 conn(0x7fcebc0a98f0 legacy=0x7fcebc137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:40:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.214+0000 7fcec0c10640 1 --1- 172.21.15.44:0/139708705 >> v1:172.21.15.44:6789/0 conn(0x7fcebc0a98f0 0x7fcebc137710 :-1 s=OPENED pgs=7460 cs=1 l=1).handle_message read tag failed 2024-03-25T17:40:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.214+0000 7fcec0c10640 1 --1- 172.21.15.44:0/139708705 >> v1:172.21.15.44:6789/0 conn(0x7fcebc0a98f0 0x7fcebc137710 :-1 s=OPENED pgs=7460 cs=1 l=1).fault on lossy channel, failing 2024-03-25T17:40:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.214+0000 7fceb97fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.44:6789/0 2024-03-25T17:40:39.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1147349977 0 0) 0x7fcebc137e20 con 0x7fcebc13f710 2024-03-25T17:40:39.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 >> v1:172.21.15.111:6789/0 conn(0x7fcebc1430c0 legacy=0x7fcebc1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:39.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 >> v1:172.21.15.44:6789/0 conn(0x7fcebc0a98f0 legacy=0x7fcebc137710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:40:39.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 10 monclient: _finish_hunting 0 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 1 monclient: found mon.c 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc138040 con 0x7fcebc13f710 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 10 monclient: _finish_auth 0 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 10 monclient: _check_auth_tickets 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fceb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 5 monclient: authenticate success, global_id 4292 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 10 monclient: _renew_subs 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc1382f0 con 0x7fcebc13f710 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 10 monclient: _renew_subs 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc1e2fb0 con 0x7fcebc13f710 2024-03-25T17:40:39.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.217+0000 7fcec2e74640 1 librados: init done 2024-03-25T17:40:39.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.218+0000 7fcec2e74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:39.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.218+0000 7fcec2e74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:39.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.218+0000 7fcec2e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:39.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.218+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc0b3ef0 con 0x7fcebc13f710 2024-03-25T17:40:39.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.543+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcea8001d60 con 0x7fcebc13f710 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.543+0000 7fceb97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.543+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fcea80037b0 con 0x7fcebc13f710 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.543+0000 7fceb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.543+0000 7fceb97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.543+0000 7fceb97fa640 10 monclient: dump: 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:39.544 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:39.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.543+0000 7fcebaffd640 4 set_mon_vals no callback set 2024-03-25T17:40:39.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.544+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fcea80197c0 con 0x7fcebc13f710 2024-03-25T17:40:39.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.544+0000 7fcec0c10640 10 monclient: get_auth_request con 0x7fcea006e080 auth_method 0 2024-03-25T17:40:39.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.544+0000 7fceb97fa640 10 client.4292.objecter ms_handle_connect 0x7fcea006e080 2024-03-25T17:40:39.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.550+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fcea8053520 con 0x7fcebc13f710 2024-03-25T17:40:39.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.550+0000 7fceb97fa640 10 client.4292.objecter ms_dispatch 0x7fcebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:40:39.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.550+0000 7fceb97fa640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:40:39.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.550+0000 7fceb97fa640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:40:39.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.550+0000 7fceb97fa640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T17:40:39.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.550+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcea804ee60 con 0x7fcebc13f710 2024-03-25T17:40:39.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.550+0000 7fceb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:39.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.550+0000 7fceb97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:39.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.806+0000 7fcec2e74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:39.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.806+0000 7fcec2e74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:39.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.806+0000 7fcec2e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:40:39.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.806+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcebc0627b0 con 0x7fcebc13f710 2024-03-25T17:40:39.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.843+0000 7fceb97fa640 1 -- 172.21.15.44:0/139708705 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fcea8050f60 con 0x7fcebc13f710 2024-03-25T17:40:39.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.843+0000 7fceb97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:39.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.843+0000 7fceb97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:39.844 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:40:39.844 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:39.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.845+0000 7fcec2e74640 10 librados: watch_flush enter 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.845+0000 7fcec2e74640 10 librados: watch_flush exit 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.845+0000 7fcec2e74640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.845+0000 7fcec2e74640 10 client.4292.objecter successfully canceled tick 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.845+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 >> v1:172.21.15.44:6800/3257043132 conn(0x7fcea006e080 legacy=0x7fcea0070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.845+0000 7fcec2e74640 10 monclient: shutdown 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.845+0000 7fcec2e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.845+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 >> v1:172.21.15.44:6790/0 conn(0x7fcebc13f710 legacy=0x7fcebc1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.846+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 shutdown_connections 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.846+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 >> 172.21.15.44:0/139708705 conn(0x7fcebc0b7ea0 msgr2=0x7fcebc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.846+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 shutdown_connections 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.846+0000 7fcec2e74640 1 -- 172.21.15.44:0/139708705 wait complete. 2024-03-25T17:40:39.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:39.846+0000 7fcec2e74640 1 librados: shutdown 2024-03-25T17:40:39.858 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:46.664 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:46.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.782+0000 7f023cb67640 10 monclient: get_monmap_and_config 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.782+0000 7f023cb67640 10 monclient: build_initial_monmap 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.782+0000 7f023cb67640 10 monclient: monmap: 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:40:46.783398+0000 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:40:46.783398+0000 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:40:46.783 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.783+0000 7f023cb67640 1 Processor -- start 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.783+0000 7f023cb67640 1 -- start start 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.783+0000 7f023cb67640 10 monclient: init 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 10 monclient(hunting): picked mon.noname-b con 0x7f02380b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 10 monclient(hunting): picked mon.noname-a con 0x7f02380b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 10 monclient(hunting): picked mon.noname-c con 0x7f023814be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02380af940 con 0x7f02380b1730 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02380b0a70 con 0x7f023814be90 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02380b0d50 con 0x7f02380b4d60 2024-03-25T17:40:46.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:46.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.784+0000 7f023cb67640 10 monclient(hunting): authenticate will time out at 6050.011949s 2024-03-25T17:40:46.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.793+0000 7f023659c640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f02380b4d60 0x7f023813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35206/0 (socket says 172.21.15.44:35206) 2024-03-25T17:40:46.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.793+0000 7f023659c640 1 -- 172.21.15.44:0/1748460984 learned_addr learned my addr 172.21.15.44:0/1748460984 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:46.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.799+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671640483 0 0) 0x7f02380b0a70 con 0x7f023814be90 2024-03-25T17:40:46.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.799+0000 7f023559a640 10 monclient(hunting): _negotiate 2024-03-25T17:40:46.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.799+0000 7f023559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:46.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.799+0000 7f023559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:46.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.799+0000 7f023559a640 10 monclient(hunting): my global_id is 4294 2024-03-25T17:40:46.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.799+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0220003520 con 0x7f023814be90 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 527796551 0 0) 0x7f0220003520 con 0x7f023814be90 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 >> v1:172.21.15.44:6790/0 conn(0x7f02380b4d60 legacy=0x7f023813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 >> v1:172.21.15.44:6789/0 conn(0x7f02380b1730 legacy=0x7f02380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 10 monclient: _finish_hunting 0 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 1 monclient: found mon.noname-c 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02380b1030 con 0x7f023814be90 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 10 monclient: _finish_auth 0 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 10 monclient: _check_auth_tickets 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0230001a10 con 0x7f023814be90 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:46.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023559a640 10 monclient: dump: 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023cb67640 5 monclient: authenticate success, global_id 4294 2024-03-25T17:40:46.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.812+0000 7f023cb67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.830+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0230001b70 con 0x7f023814be90 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.830+0000 7f023559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.830+0000 7f023559a640 1 -- 172.21.15.44:0/1748460984 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0230001f80 con 0x7f023814be90 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.830+0000 7f023559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.830+0000 7f023559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.830+0000 7f023559a640 10 monclient: dump: 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:46.831 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.831+0000 7f023cb67640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:46.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.831+0000 7f023cb67640 4 set_mon_vals no callback set 2024-03-25T17:40:46.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.831+0000 7f023cb67640 10 monclient: shutdown 2024-03-25T17:40:46.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.831+0000 7f023cb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.831+0000 7f023cb67640 1 -- 172.21.15.44:0/1748460984 >> v1:172.21.15.111:6789/0 conn(0x7f023814be90 legacy=0x7f023814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.832+0000 7f023cb67640 1 -- 172.21.15.44:0/1748460984 shutdown_connections 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.832+0000 7f023cb67640 1 -- 172.21.15.44:0/1748460984 >> 172.21.15.44:0/1748460984 conn(0x7f02380617a0 msgr2=0x7f0238061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.832+0000 7f023cb67640 1 -- 172.21.15.44:0/1748460984 shutdown_connections 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.832+0000 7f023cb67640 1 -- 172.21.15.44:0/1748460984 wait complete. 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 10 monclient: build_initial_monmap 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 10 monclient: monmap: 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:40:46.834 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 1 librados: starting msgr at 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 1 librados: starting objecter 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 1 Processor -- start 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 1 -- start start 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 1 librados: setting wanted keys 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 1 librados: calling monclient init 2024-03-25T17:40:46.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.833+0000 7f023cb67640 10 monclient: init 2024-03-25T17:40:46.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:46.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:46.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 10 monclient(hunting): picked mon.noname-a con 0x7f02380b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:46.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 10 monclient(hunting): picked mon.noname-c con 0x7f02380b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:46.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 10 monclient(hunting): picked mon.noname-b con 0x7f023814be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:46.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0238140380 con 0x7f02380b1730 2024-03-25T17:40:46.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0238140660 con 0x7f023814be90 2024-03-25T17:40:46.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0238141e30 con 0x7f02380b4d60 2024-03-25T17:40:46.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:46.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.834+0000 7f023cb67640 10 monclient(hunting): authenticate will time out at 6050.062319s 2024-03-25T17:40:46.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.843+0000 7f0236d9d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f023814be90 0x7f02381e2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:51268/0 (socket says 172.21.15.44:51268) 2024-03-25T17:40:46.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.843+0000 7f0236d9d640 1 -- 172.21.15.44:0/1919740586 learned_addr learned my addr 172.21.15.44:0/1919740586 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:46.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.845+0000 7f0226ffd640 10 client.?.objecter ms_handle_connect 0x7f023814be90 2024-03-25T17:40:46.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.845+0000 7f0226ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:46.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.847+0000 7f0226ffd640 10 client.?.objecter ms_handle_connect 0x7f02380b4d60 2024-03-25T17:40:46.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.847+0000 7f0226ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:46.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.856+0000 7f0226ffd640 10 client.?.objecter ms_handle_connect 0x7f02380b1730 2024-03-25T17:40:46.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.856+0000 7f0226ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:46.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.857+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243483528 0 0) 0x7f0238140660 con 0x7f023814be90 2024-03-25T17:40:46.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.857+0000 7f0226ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:46.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.857+0000 7f0226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:46.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.857+0000 7f0226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:46.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.857+0000 7f0226ffd640 10 monclient(hunting): my global_id is 4297 2024-03-25T17:40:46.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.857+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0218004eb0 con 0x7f023814be90 2024-03-25T17:40:46.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.859+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045773891 0 0) 0x7f0238141e30 con 0x7f02380b4d60 2024-03-25T17:40:46.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.859+0000 7f0226ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:46.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.859+0000 7f0226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:46.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.859+0000 7f0226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:46.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.859+0000 7f0226ffd640 10 monclient(hunting): my global_id is 4298 2024-03-25T17:40:46.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.859+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0238140660 con 0x7f02380b4d60 2024-03-25T17:40:46.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.869+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089304132 0 0) 0x7f0238140380 con 0x7f02380b1730 2024-03-25T17:40:46.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.869+0000 7f0226ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:46.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.869+0000 7f0226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:46.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.869+0000 7f0226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:46.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.869+0000 7f0226ffd640 10 monclient(hunting): my global_id is 4311 2024-03-25T17:40:46.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.869+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0238141e30 con 0x7f02380b1730 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.871+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2366057318 0 0) 0x7f0218004eb0 con 0x7f023814be90 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.871+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0238140380 con 0x7f023814be90 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.871+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0230001f40 con 0x7f023814be90 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.871+0000 7f0226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.871+0000 7f0226ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.871+0000 7f0226ffd640 10 monclient(hunting): dump: 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:46.872 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:46.873 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:46.873 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:46.873 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:46.873 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:46.873 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:46.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.871+0000 7f0226ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:46.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.886+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892164088 0 0) 0x7f0238140660 con 0x7f02380b4d60 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.886+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0218004eb0 con 0x7f02380b4d60 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.886+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f022c0018b0 con 0x7f02380b4d60 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.886+0000 7f0226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.886+0000 7f0226ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.886+0000 7f0226ffd640 10 monclient(hunting): dump: 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:46.888 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1787978159 0 0) 0x7f0238140380 con 0x7f023814be90 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 >> v1:172.21.15.44:6790/0 conn(0x7f02380b4d60 legacy=0x7f0238142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 >> v1:172.21.15.44:6789/0 conn(0x7f02380b1730 legacy=0x7f0238138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 1 monclient: found mon.b 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238142110 con 0x7f023814be90 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 10 monclient: _finish_auth 0 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f0226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 5 monclient: authenticate success, global_id 4297 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 10 monclient: _renew_subs 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:46.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0238140e60 con 0x7f023814be90 2024-03-25T17:40:46.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:46.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 10 monclient: _renew_subs 2024-03-25T17:40:46.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:46.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0238142c40 con 0x7f023814be90 2024-03-25T17:40:46.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.887+0000 7f023cb67640 1 librados: init done 2024-03-25T17:40:46.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.888+0000 7f023cb67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:46.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.888+0000 7f023cb67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:46.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.888+0000 7f023cb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:46.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:46.888+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02380627b0 con 0x7f023814be90 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.163+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0230003c20 con 0x7f023814be90 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.163+0000 7f0226ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.163+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0230003fb0 con 0x7f023814be90 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.163+0000 7f0226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.163+0000 7f0226ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.163+0000 7f0226ffd640 10 monclient: dump: 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:47.164 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:47.165 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:47.165 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:47.165 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:47.165 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:47.165 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:47.165 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:47.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.163+0000 7f0234d99640 4 set_mon_vals no callback set 2024-03-25T17:40:47.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.164+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f02300041d0 con 0x7f023814be90 2024-03-25T17:40:47.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.165+0000 7f023659c640 10 monclient: get_auth_request con 0x7f021806df70 auth_method 0 2024-03-25T17:40:47.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.165+0000 7f0226ffd640 10 client.4297.objecter ms_handle_connect 0x7f021806df70 2024-03-25T17:40:47.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.170+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f0230054ab0 con 0x7f023814be90 2024-03-25T17:40:47.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.170+0000 7f0226ffd640 10 client.4297.objecter ms_dispatch 0x7f0238000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:40:47.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.170+0000 7f0226ffd640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:40:47.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.170+0000 7f0226ffd640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:40:47.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.170+0000 7f0226ffd640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-25T17:40:47.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.171+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02300505c0 con 0x7f023814be90 2024-03-25T17:40:47.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.171+0000 7f0226ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:47.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.171+0000 7f0226ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:47.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.445+0000 7f023cb67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:47.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.445+0000 7f023cb67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:47.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.445+0000 7f023cb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:40:47.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.445+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0238140810 con 0x7f023814be90 2024-03-25T17:40:47.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.474+0000 7f0226ffd640 1 -- 172.21.15.44:0/1919740586 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f02300524f0 con 0x7f023814be90 2024-03-25T17:40:47.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.474+0000 7f0226ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:47.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.474+0000 7f0226ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:47.475 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:40:47.475 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:47.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 10 librados: watch_flush enter 2024-03-25T17:40:47.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 10 librados: watch_flush exit 2024-03-25T17:40:47.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-25T17:40:47.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 10 client.4297.objecter successfully canceled tick 2024-03-25T17:40:47.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 >> v1:172.21.15.44:6800/3257043132 conn(0x7f021806df70 legacy=0x7f0218070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:47.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 10 monclient: shutdown 2024-03-25T17:40:47.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:47.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 >> v1:172.21.15.111:6789/0 conn(0x7f023814be90 legacy=0x7f02381e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:47.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 shutdown_connections 2024-03-25T17:40:47.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 >> 172.21.15.44:0/1919740586 conn(0x7f02380617a0 msgr2=0x7f02380b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:47.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 shutdown_connections 2024-03-25T17:40:47.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 1 -- 172.21.15.44:0/1919740586 wait complete. 2024-03-25T17:40:47.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:47.476+0000 7f023cb67640 1 librados: shutdown 2024-03-25T17:40:47.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:54.296 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:54.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.413+0000 7f0ee958f640 10 monclient: get_monmap_and_config 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.413+0000 7f0ee958f640 10 monclient: build_initial_monmap 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.414+0000 7f0ee958f640 10 monclient: monmap: 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:40:54.414903+0000 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:40:54.414903+0000 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.414+0000 7f0ee958f640 1 Processor -- start 2024-03-25T17:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.414+0000 7f0ee958f640 1 -- start start 2024-03-25T17:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.414+0000 7f0ee958f640 10 monclient: init 2024-03-25T17:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee40a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee413f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee41430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee4142520 con 0x7f0ee41430c0 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee40b28a0 con 0x7f0ee40a98f0 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee40b2a30 con 0x7f0ee413f710 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.415+0000 7f0ee958f640 10 monclient(hunting): authenticate will time out at 6057.643134s 2024-03-25T17:40:54.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.421+0000 7f0ee27fc640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f0ee413f710 0x7f0ee41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:59874/0 (socket says 172.21.15.44:59874) 2024-03-25T17:40:54.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.421+0000 7f0ee27fc640 1 -- 172.21.15.44:0/3144186774 learned_addr learned my addr 172.21.15.44:0/3144186774 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:54.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.436+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931862062 0 0) 0x7f0ee40b2a30 con 0x7f0ee413f710 2024-03-25T17:40:54.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.437+0000 7f0ee1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:54.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.437+0000 7f0ee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:54.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.437+0000 7f0ee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:54.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.437+0000 7f0ee1ffb640 10 monclient(hunting): my global_id is 4301 2024-03-25T17:40:54.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.437+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ecc003520 con 0x7f0ee413f710 2024-03-25T17:40:54.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.439+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358964017 0 0) 0x7f0ee40b28a0 con 0x7f0ee40a98f0 2024-03-25T17:40:54.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.439+0000 7f0ee1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:54.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.439+0000 7f0ee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:54.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.439+0000 7f0ee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:54.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.439+0000 7f0ee1ffb640 10 monclient(hunting): my global_id is 4300 2024-03-25T17:40:54.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.439+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee40b2a30 con 0x7f0ee40a98f0 2024-03-25T17:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.441+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717979451 0 0) 0x7f0ee4142520 con 0x7f0ee41430c0 2024-03-25T17:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.441+0000 7f0ee1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.441+0000 7f0ee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:54.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.441+0000 7f0ee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:54.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.441+0000 7f0ee1ffb640 10 monclient(hunting): my global_id is 4314 2024-03-25T17:40:54.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.441+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee40b28a0 con 0x7f0ee41430c0 2024-03-25T17:40:54.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1304886803 0 0) 0x7f0ecc003520 con 0x7f0ee413f710 2024-03-25T17:40:54.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 >> v1:172.21.15.111:6789/0 conn(0x7f0ee40a98f0 legacy=0x7f0ee40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:54.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 >> v1:172.21.15.44:6789/0 conn(0x7f0ee41430c0 legacy=0x7f0ee414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:54.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:40:54.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 1 monclient: found mon.noname-b 2024-03-25T17:40:54.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee40b44c0 con 0x7f0ee413f710 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0ed8001690 con 0x7f0ee413f710 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee1ffb640 10 monclient: dump: 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.457 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:54.458 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:54.458 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:54.458 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:54.458 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:54.458 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:54.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee958f640 5 monclient: authenticate success, global_id 4301 2024-03-25T17:40:54.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.455+0000 7f0ee958f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:54.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.469+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ed80017f0 con 0x7f0ee413f710 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.469+0000 7f0ee1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.469+0000 7f0ee1ffb640 1 -- 172.21.15.44:0/3144186774 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0ed8001c00 con 0x7f0ee413f710 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.469+0000 7f0ee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.469+0000 7f0ee1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.469+0000 7f0ee1ffb640 10 monclient: dump: 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:54.470 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:54.471 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:54.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.469+0000 7f0ee958f640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:54.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.469+0000 7f0ee958f640 4 set_mon_vals no callback set 2024-03-25T17:40:54.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.470+0000 7f0ee958f640 10 monclient: shutdown 2024-03-25T17:40:54.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.470+0000 7f0ee958f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:54.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.470+0000 7f0ee958f640 1 -- 172.21.15.44:0/3144186774 >> v1:172.21.15.44:6790/0 conn(0x7f0ee413f710 legacy=0x7f0ee41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:54.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.470+0000 7f0ee958f640 1 -- 172.21.15.44:0/3144186774 shutdown_connections 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.470+0000 7f0ee958f640 1 -- 172.21.15.44:0/3144186774 >> 172.21.15.44:0/3144186774 conn(0x7f0ee40b7ea0 msgr2=0x7f0ee40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.470+0000 7f0ee958f640 1 -- 172.21.15.44:0/3144186774 shutdown_connections 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.470+0000 7f0ee958f640 1 -- 172.21.15.44:0/3144186774 wait complete. 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.471+0000 7f0ee958f640 10 monclient: build_initial_monmap 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.471+0000 7f0ee958f640 10 monclient: monmap: 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:40:54.473 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:54.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.471+0000 7f0ee958f640 1 librados: starting msgr at 2024-03-25T17:40:54.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.471+0000 7f0ee958f640 1 librados: starting objecter 2024-03-25T17:40:54.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.472+0000 7f0ee958f640 1 Processor -- start 2024-03-25T17:40:54.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.472+0000 7f0ee958f640 1 -- start start 2024-03-25T17:40:54.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.472+0000 7f0ee958f640 1 librados: setting wanted keys 2024-03-25T17:40:54.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.472+0000 7f0ee958f640 1 librados: calling monclient init 2024-03-25T17:40:54.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.472+0000 7f0ee958f640 10 monclient: init 2024-03-25T17:40:54.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.472+0000 7f0ee958f640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.472+0000 7f0ee958f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.472+0000 7f0ee958f640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee40a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.473+0000 7f0ee958f640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee413f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.473+0000 7f0ee958f640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee41430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.473+0000 7f0ee958f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee41370a0 con 0x7f0ee41430c0 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.473+0000 7f0ee958f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee4135090 con 0x7f0ee40a98f0 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.473+0000 7f0ee958f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee4135220 con 0x7f0ee413f710 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.473+0000 7f0ee958f640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.473+0000 7f0ee958f640 10 monclient(hunting): authenticate will time out at 6057.701132s 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.475+0000 7f0ee2ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0ee40a98f0 0x7f0ee41dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:35740/0 (socket says 172.21.15.44:35740) 2024-03-25T17:40:54.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.475+0000 7f0ee2ffd640 1 -- 172.21.15.44:0/1833046730 learned_addr learned my addr 172.21.15.44:0/1833046730 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:40:54.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.479+0000 7f0ecb7fe640 10 client.?.objecter ms_handle_connect 0x7f0ee413f710 2024-03-25T17:40:54.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.479+0000 7f0ecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:54.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.480+0000 7f0ecb7fe640 10 client.?.objecter ms_handle_connect 0x7f0ee40a98f0 2024-03-25T17:40:54.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.480+0000 7f0ecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:54.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.487+0000 7f0ecb7fe640 10 client.?.objecter ms_handle_connect 0x7f0ee41430c0 2024-03-25T17:40:54.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.487+0000 7f0ecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:54.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.492+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119296168 0 0) 0x7f0ee4135090 con 0x7f0ee40a98f0 2024-03-25T17:40:54.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.492+0000 7f0ecb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:40:54.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.492+0000 7f0ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:54.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.492+0000 7f0ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:54.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.492+0000 7f0ecb7fe640 10 monclient(hunting): my global_id is 4303 2024-03-25T17:40:54.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.492+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed0004eb0 con 0x7f0ee40a98f0 2024-03-25T17:40:54.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.499+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277541278 0 0) 0x7f0ee41370a0 con 0x7f0ee41430c0 2024-03-25T17:40:54.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.499+0000 7f0ecb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:40:54.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.499+0000 7f0ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:54.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.499+0000 7f0ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:54.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.499+0000 7f0ecb7fe640 10 monclient(hunting): my global_id is 4317 2024-03-25T17:40:54.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.499+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed00051d0 con 0x7f0ee41430c0 2024-03-25T17:40:54.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.508+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636357038 0 0) 0x7f0ee4135220 con 0x7f0ee413f710 2024-03-25T17:40:54.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.508+0000 7f0ecb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:40:54.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.508+0000 7f0ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:54.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.508+0000 7f0ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:54.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.508+0000 7f0ecb7fe640 10 monclient(hunting): my global_id is 4304 2024-03-25T17:40:54.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.508+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee41370a0 con 0x7f0ee413f710 2024-03-25T17:40:54.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.514+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1766826759 0 0) 0x7f0ed0004eb0 con 0x7f0ee40a98f0 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.514+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ee4135220 con 0x7f0ee40a98f0 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.514+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0ed40019f0 con 0x7f0ee40a98f0 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.514+0000 7f0ecb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.514+0000 7f0ecb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.514+0000 7f0ecb7fe640 10 monclient(hunting): dump: 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:54.515 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:54.516 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:54.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.514+0000 7f0ecb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.523+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1040154811 0 0) 0x7f0ed00051d0 con 0x7f0ee41430c0 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.523+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ed0004eb0 con 0x7f0ee41430c0 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.523+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0edc001b50 con 0x7f0ee41430c0 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.523+0000 7f0ecb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.523+0000 7f0ecb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.523+0000 7f0ecb7fe640 10 monclient(hunting): dump: 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:54.524 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:54.525 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:54.525 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:54.525 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:54.525 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:54.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.534+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3902730769 0 0) 0x7f0ee41370a0 con 0x7f0ee413f710 2024-03-25T17:40:54.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.534+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ed00051d0 con 0x7f0ee413f710 2024-03-25T17:40:54.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.534+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0ed8001b00 con 0x7f0ee413f710 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.534+0000 7f0ecb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.534+0000 7f0ecb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.534+0000 7f0ecb7fe640 10 monclient(hunting): dump: 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:54.535 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:54.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 570117633 0 0) 0x7f0ed0004eb0 con 0x7f0ee41430c0 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 >> v1:172.21.15.44:6790/0 conn(0x7f0ee413f710 legacy=0x7f0ee41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 >> v1:172.21.15.111:6789/0 conn(0x7f0ee40a98f0 legacy=0x7f0ee41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 1 monclient: found mon.a 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee41d5000 con 0x7f0ee41430c0 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 10 monclient: _finish_auth 0 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 5 monclient: authenticate success, global_id 4317 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 10 monclient: _renew_subs 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:54.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee41d52b0 con 0x7f0ee41430c0 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 10 monclient: _renew_subs 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee4135910 con 0x7f0ee41430c0 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.537+0000 7f0ee958f640 1 librados: init done 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.538+0000 7f0ee958f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.538+0000 7f0ee958f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.538+0000 7f0ee958f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:54.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.538+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee40627b0 con 0x7f0ee41430c0 2024-03-25T17:40:54.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.897+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0edc001fe0 con 0x7f0ee41430c0 2024-03-25T17:40:54.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.897+0000 7f0ecb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:54.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.897+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0edc003b30 con 0x7f0ee41430c0 2024-03-25T17:40:54.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.897+0000 7f0ecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:54.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.897+0000 7f0ecb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:54.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.897+0000 7f0ecb7fe640 10 monclient: dump: 2024-03-25T17:40:54.898 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:40:54.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.897+0000 7f0ee17fa640 4 set_mon_vals no callback set 2024-03-25T17:40:54.900 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.898+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f0edc019b00 con 0x7f0ee41430c0 2024-03-25T17:40:54.900 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.899+0000 7f0ee2ffd640 10 monclient: get_auth_request con 0x7f0ed006e2e0 auth_method 0 2024-03-25T17:40:54.900 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.899+0000 7f0ecb7fe640 10 client.4317.objecter ms_handle_connect 0x7f0ed006e2e0 2024-03-25T17:40:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.905+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f0edc053880 con 0x7f0ee41430c0 2024-03-25T17:40:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.905+0000 7f0ecb7fe640 10 client.4317.objecter ms_dispatch 0x7f0ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:40:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.905+0000 7f0ecb7fe640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:40:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.905+0000 7f0ecb7fe640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:40:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.905+0000 7f0ecb7fe640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T17:40:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.905+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0edc04f1c0 con 0x7f0ee41430c0 2024-03-25T17:40:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.905+0000 7f0ecb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:54.905+0000 7f0ecb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:55.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.165+0000 7f0ee958f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:55.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.165+0000 7f0ee958f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:55.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.165+0000 7f0ee958f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:40:55.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.165+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ee40008d0 con 0x7f0ee41430c0 2024-03-25T17:40:55.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.194+0000 7f0ecb7fe640 1 -- 172.21.15.44:0/1833046730 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f0edc0512c0 con 0x7f0ee41430c0 2024-03-25T17:40:55.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.194+0000 7f0ecb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:55.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.194+0000 7f0ecb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:55.195 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:40:55.195 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:55.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 10 librados: watch_flush enter 2024-03-25T17:40:55.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 10 librados: watch_flush exit 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 10 client.4317.objecter successfully canceled tick 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 >> v1:172.21.15.44:6800/3257043132 conn(0x7f0ed006e2e0 legacy=0x7f0ed00707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 10 monclient: shutdown 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 >> v1:172.21.15.44:6789/0 conn(0x7f0ee41430c0 legacy=0x7f0ee41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 shutdown_connections 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 >> 172.21.15.44:0/1833046730 conn(0x7f0ee40b7ea0 msgr2=0x7f0ee4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 shutdown_connections 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 1 -- 172.21.15.44:0/1833046730 wait complete. 2024-03-25T17:40:55.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:40:55.197+0000 7f0ee958f640 1 librados: shutdown 2024-03-25T17:40:55.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:57.294 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:14.649 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:57.356 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:14.712 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:57.390 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:14.746 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:57.397 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:14.753 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:57.406 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:14.762 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:57.411 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:14.731 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:57.422 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:14.742 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:57.429 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:14.785 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:57.440 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:14.761 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:02.017 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:41:02.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.119+0000 7faf76ac0640 10 monclient: get_monmap_and_config 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.119+0000 7faf76ac0640 10 monclient: build_initial_monmap 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.119+0000 7faf76ac0640 10 monclient: monmap: 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:41:02.120335+0000 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:41:02.120335+0000 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:02.120 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:41:02.121 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:41:02.121 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 1 Processor -- start 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 1 -- start start 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 10 monclient: init 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 10 monclient(hunting): picked mon.noname-b con 0x7faf700b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 10 monclient(hunting): picked mon.noname-a con 0x7faf700b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 10 monclient(hunting): picked mon.noname-c con 0x7faf7014be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700af940 con 0x7faf700b1730 2024-03-25T17:41:02.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700b0a70 con 0x7faf7014be90 2024-03-25T17:41:02.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700b0d50 con 0x7faf700b4d60 2024-03-25T17:41:02.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:02.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.120+0000 7faf76ac0640 10 monclient(hunting): authenticate will time out at 6065.348493s 2024-03-25T17:41:02.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.121+0000 7faf7485c640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7faf700b4d60 0x7faf7013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:42858/0 (socket says 172.21.15.44:42858) 2024-03-25T17:41:02.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.121+0000 7faf7485c640 1 -- 172.21.15.44:0/4157754338 learned_addr learned my addr 172.21.15.44:0/4157754338 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:02.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.145+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297584707 0 0) 0x7faf700af940 con 0x7faf700b1730 2024-03-25T17:41:02.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.145+0000 7faf677fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:02.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.145+0000 7faf677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:02.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.145+0000 7faf677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:02.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.145+0000 7faf677fe640 10 monclient(hunting): my global_id is 4320 2024-03-25T17:41:02.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.145+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf58003520 con 0x7faf700b1730 2024-03-25T17:41:02.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 561086054 0 0) 0x7faf58003520 con 0x7faf700b1730 2024-03-25T17:41:02.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 >> v1:172.21.15.44:6790/0 conn(0x7faf700b4d60 legacy=0x7faf7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:02.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 >> v1:172.21.15.111:6789/0 conn(0x7faf7014be90 legacy=0x7faf7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:02.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 10 monclient: _finish_hunting 0 2024-03-25T17:41:02.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 1 monclient: found mon.noname-a 2024-03-25T17:41:02.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:02.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700b1030 con 0x7faf700b1730 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 10 monclient: _finish_auth 0 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 10 monclient: _check_auth_tickets 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7faf60001690 con 0x7faf700b1730 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf677fe640 10 monclient: dump: 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:02.172 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:02.173 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:02.173 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:02.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf76ac0640 5 monclient: authenticate success, global_id 4320 2024-03-25T17:41:02.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.170+0000 7faf76ac0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:02.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.194+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf600017f0 con 0x7faf700b1730 2024-03-25T17:41:02.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.194+0000 7faf677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:02.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.194+0000 7faf677fe640 1 -- 172.21.15.44:0/4157754338 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7faf60001c00 con 0x7faf700b1730 2024-03-25T17:41:02.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.194+0000 7faf677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:02.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.194+0000 7faf677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:41:02.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.194+0000 7faf677fe640 10 monclient: dump: 2024-03-25T17:41:02.195 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:02.195 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 10 monclient: get_monmap_and_config success 2024-03-25T17:41:02.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 4 set_mon_vals no callback set 2024-03-25T17:41:02.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 10 monclient: shutdown 2024-03-25T17:41:02.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:02.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 1 -- 172.21.15.44:0/4157754338 >> v1:172.21.15.44:6789/0 conn(0x7faf700b1730 legacy=0x7faf700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:02.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 1 -- 172.21.15.44:0/4157754338 shutdown_connections 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 1 -- 172.21.15.44:0/4157754338 >> 172.21.15.44:0/4157754338 conn(0x7faf700617a0 msgr2=0x7faf70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 1 -- 172.21.15.44:0/4157754338 shutdown_connections 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.195+0000 7faf76ac0640 1 -- 172.21.15.44:0/4157754338 wait complete. 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.196+0000 7faf76ac0640 10 monclient: build_initial_monmap 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.196+0000 7faf76ac0640 10 monclient: monmap: 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:41:02.199 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.196+0000 7faf76ac0640 1 librados: starting msgr at 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.196+0000 7faf76ac0640 1 librados: starting objecter 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 1 Processor -- start 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 1 -- start start 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 1 librados: setting wanted keys 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 1 librados: calling monclient init 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 10 monclient: init 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 10 monclient(hunting): picked mon.noname-c con 0x7faf700b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.197+0000 7faf76ac0640 10 monclient(hunting): picked mon.noname-a con 0x7faf700b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.198+0000 7faf76ac0640 10 monclient(hunting): picked mon.noname-b con 0x7faf7014be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.198+0000 7faf76ac0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf70138b50 con 0x7faf700b4d60 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.198+0000 7faf76ac0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf70142260 con 0x7faf7014be90 2024-03-25T17:41:02.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.198+0000 7faf76ac0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf701d79f0 con 0x7faf700b1730 2024-03-25T17:41:02.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.198+0000 7faf76ac0640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:02.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.198+0000 7faf76ac0640 10 monclient(hunting): authenticate will time out at 6065.425723s 2024-03-25T17:41:02.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.214+0000 7faf67fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7faf700b4d60 0x7faf701d7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:38668/0 (socket says 172.21.15.44:38668) 2024-03-25T17:41:02.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.214+0000 7faf67fff640 1 -- 172.21.15.44:0/2373019289 learned_addr learned my addr 172.21.15.44:0/2373019289 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:02.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.217+0000 7faf657fa640 10 client.?.objecter ms_handle_connect 0x7faf7014be90 2024-03-25T17:41:02.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.217+0000 7faf657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:02.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.222+0000 7faf657fa640 10 client.?.objecter ms_handle_connect 0x7faf700b1730 2024-03-25T17:41:02.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.222+0000 7faf657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:02.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.223+0000 7faf657fa640 10 client.?.objecter ms_handle_connect 0x7faf700b4d60 2024-03-25T17:41:02.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.223+0000 7faf657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:02.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.229+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040407570 0 0) 0x7faf70142260 con 0x7faf7014be90 2024-03-25T17:41:02.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.229+0000 7faf657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:41:02.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.229+0000 7faf657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:02.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.229+0000 7faf657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:02.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.229+0000 7faf657fa640 10 monclient(hunting): my global_id is 4309 2024-03-25T17:41:02.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.229+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf5c004eb0 con 0x7faf7014be90 2024-03-25T17:41:02.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.231+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395385185 0 0) 0x7faf70138b50 con 0x7faf700b4d60 2024-03-25T17:41:02.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.231+0000 7faf657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:41:02.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.231+0000 7faf657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:02.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.231+0000 7faf657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:02.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.231+0000 7faf657fa640 10 monclient(hunting): my global_id is 4323 2024-03-25T17:41:02.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.231+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf70142260 con 0x7faf700b4d60 2024-03-25T17:41:02.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.241+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925361469 0 0) 0x7faf701d79f0 con 0x7faf700b1730 2024-03-25T17:41:02.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.241+0000 7faf657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:41:02.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.241+0000 7faf657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:02.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.241+0000 7faf657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:02.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.241+0000 7faf657fa640 10 monclient(hunting): my global_id is 4310 2024-03-25T17:41:02.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.241+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf70138b50 con 0x7faf700b1730 2024-03-25T17:41:02.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.245+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2248572874 0 0) 0x7faf5c004eb0 con 0x7faf7014be90 2024-03-25T17:41:02.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.245+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf701d79f0 con 0x7faf7014be90 2024-03-25T17:41:02.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.245+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7faf680017a0 con 0x7faf7014be90 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.245+0000 7faf657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.245+0000 7faf657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.245+0000 7faf657fa640 10 monclient(hunting): dump: 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:02.246 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:02.247 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:02.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.245+0000 7faf657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:41:02.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.250+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2421552437 0 0) 0x7faf70142260 con 0x7faf700b4d60 2024-03-25T17:41:02.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.250+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf5c004eb0 con 0x7faf700b4d60 2024-03-25T17:41:02.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.250+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7faf60001c00 con 0x7faf700b4d60 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.250+0000 7faf657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.250+0000 7faf657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.250+0000 7faf657fa640 10 monclient(hunting): dump: 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:02.251 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:02.252 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:02.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 692300989 0 0) 0x7faf5c004eb0 con 0x7faf700b4d60 2024-03-25T17:41:02.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 >> v1:172.21.15.44:6790/0 conn(0x7faf700b1730 legacy=0x7faf70138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:02.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 >> v1:172.21.15.111:6789/0 conn(0x7faf7014be90 legacy=0x7faf70142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:02.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 10 monclient: _finish_hunting 0 2024-03-25T17:41:02.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 1 monclient: found mon.a 2024-03-25T17:41:02.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:02.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:02.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf701d8620 con 0x7faf700b4d60 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 10 monclient: _finish_auth 0 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 10 monclient: _check_auth_tickets 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 5 monclient: authenticate success, global_id 4323 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 10 monclient: _renew_subs 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf701434a0 con 0x7faf700b4d60 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 10 monclient: _renew_subs 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf701427f0 con 0x7faf700b4d60 2024-03-25T17:41:02.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.259+0000 7faf76ac0640 1 librados: init done 2024-03-25T17:41:02.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.260+0000 7faf76ac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:02.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.260+0000 7faf76ac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:02.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.260+0000 7faf76ac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:02.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.260+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf701401f0 con 0x7faf700b4d60 2024-03-25T17:41:02.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.587+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf600038a0 con 0x7faf700b4d60 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.587+0000 7faf657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.587+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7faf60003bf0 con 0x7faf700b4d60 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.587+0000 7faf657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.587+0000 7faf657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.587+0000 7faf657fa640 10 monclient: dump: 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.588 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:02.589 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:02.589 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:02.589 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:02.589 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:02.589 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:02.589 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:02.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.587+0000 7faf66ffd640 4 set_mon_vals no callback set 2024-03-25T17:41:02.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.588+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7faf60003dd0 con 0x7faf700b4d60 2024-03-25T17:41:02.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.588+0000 7faf7485c640 10 monclient: get_auth_request con 0x7faf5c06df70 auth_method 0 2024-03-25T17:41:02.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.588+0000 7faf657fa640 10 client.4323.objecter ms_handle_connect 0x7faf5c06df70 2024-03-25T17:41:02.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.594+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7faf600549e0 con 0x7faf700b4d60 2024-03-25T17:41:02.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.594+0000 7faf657fa640 10 client.4323.objecter ms_dispatch 0x7faf70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:41:02.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.594+0000 7faf657fa640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:41:02.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.594+0000 7faf657fa640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:41:02.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.595+0000 7faf657fa640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T17:41:02.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.595+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf60050320 con 0x7faf700b4d60 2024-03-25T17:41:02.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.595+0000 7faf657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:02.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.595+0000 7faf657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:41:02.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.838+0000 7faf76ac0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:02.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.838+0000 7faf76ac0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:02.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.838+0000 7faf76ac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:02.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.838+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf701d7850 con 0x7faf700b4d60 2024-03-25T17:41:02.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.862+0000 7faf657fa640 1 -- 172.21.15.44:0/2373019289 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7faf701d7850 con 0x7faf700b4d60 2024-03-25T17:41:02.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.862+0000 7faf657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:02.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.862+0000 7faf657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:41:02.863 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:41:02.863 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:41:02.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 10 librados: watch_flush enter 2024-03-25T17:41:02.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 10 librados: watch_flush exit 2024-03-25T17:41:02.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T17:41:02.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 10 client.4323.objecter successfully canceled tick 2024-03-25T17:41:02.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 >> v1:172.21.15.44:6800/3257043132 conn(0x7faf5c06df70 legacy=0x7faf5c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:02.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 10 monclient: shutdown 2024-03-25T17:41:02.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:02.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 >> v1:172.21.15.44:6789/0 conn(0x7faf700b4d60 legacy=0x7faf701d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:02.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 shutdown_connections 2024-03-25T17:41:02.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 >> 172.21.15.44:0/2373019289 conn(0x7faf700617a0 msgr2=0x7faf700b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:02.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 shutdown_connections 2024-03-25T17:41:02.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 1 -- 172.21.15.44:0/2373019289 wait complete. 2024-03-25T17:41:02.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:02.865+0000 7faf76ac0640 1 librados: shutdown 2024-03-25T17:41:02.879 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:41:09.684 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:41:09.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.798+0000 7f952af5d640 10 monclient: get_monmap_and_config 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.798+0000 7f952af5d640 10 monclient: build_initial_monmap 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.798+0000 7f952af5d640 10 monclient: monmap: 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:41:09.799403+0000 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:41:09.799403+0000 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:41:09.799 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.799+0000 7f952af5d640 1 Processor -- start 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.799+0000 7f952af5d640 1 -- start start 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.799+0000 7f952af5d640 10 monclient: init 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 10 monclient(hunting): picked mon.noname-c con 0x7f95240b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 10 monclient(hunting): picked mon.noname-b con 0x7f95240b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 10 monclient(hunting): picked mon.noname-a con 0x7f952414be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240af940 con 0x7f952414be90 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240b0a70 con 0x7f95240b4d60 2024-03-25T17:41:09.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240b0d50 con 0x7f95240b1730 2024-03-25T17:41:09.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:09.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.800+0000 7f952af5d640 10 monclient(hunting): authenticate will time out at 6073.028124s 2024-03-25T17:41:09.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.807+0000 7f9523fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f95240b1730 0x7f95240b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:42000/0 (socket says 172.21.15.44:42000) 2024-03-25T17:41:09.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.807+0000 7f9523fff640 1 -- 172.21.15.44:0/142040941 learned_addr learned my addr 172.21.15.44:0/142040941 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:09.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.820+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709975302 0 0) 0x7f95240b0d50 con 0x7f95240b1730 2024-03-25T17:41:09.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.820+0000 7f95237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:09.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.820+0000 7f95237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:09.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.820+0000 7f95237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:09.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.821+0000 7f95237fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T17:41:09.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.821+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510003520 con 0x7f95240b1730 2024-03-25T17:41:09.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.825+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055561125 0 0) 0x7f95240b0a70 con 0x7f95240b4d60 2024-03-25T17:41:09.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.825+0000 7f95237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:09.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.825+0000 7f95237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:09.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.825+0000 7f95237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:09.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.825+0000 7f95237fe640 10 monclient(hunting): my global_id is 4312 2024-03-25T17:41:09.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.825+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95240b0d50 con 0x7f95240b4d60 2024-03-25T17:41:09.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.827+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950888191 0 0) 0x7f95240af940 con 0x7f952414be90 2024-03-25T17:41:09.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.827+0000 7f95237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:09.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.827+0000 7f95237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:09.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.827+0000 7f95237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:09.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.827+0000 7f95237fe640 10 monclient(hunting): my global_id is 4326 2024-03-25T17:41:09.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.827+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95240b0a70 con 0x7f952414be90 2024-03-25T17:41:09.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004704858 0 0) 0x7f95240b0a70 con 0x7f952414be90 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 >> v1:172.21.15.44:6790/0 conn(0x7f95240b1730 legacy=0x7f95240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 >> v1:172.21.15.111:6789/0 conn(0x7f95240b4d60 legacy=0x7f952413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 10 monclient: _finish_hunting 0 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 1 monclient: found mon.noname-a 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95240b1030 con 0x7f952414be90 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 10 monclient: _finish_auth 0 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 10 monclient: _check_auth_tickets 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f951c001930 con 0x7f952414be90 2024-03-25T17:41:09.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f952af5d640 5 monclient: authenticate success, global_id 4326 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f952af5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f95237fe640 10 monclient: dump: 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:09.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.845+0000 7f952af5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:09.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.856+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f951c001a90 con 0x7f952414be90 2024-03-25T17:41:09.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.856+0000 7f95237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:09.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.856+0000 7f95237fe640 1 -- 172.21.15.44:0/142040941 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f951c001ee0 con 0x7f952414be90 2024-03-25T17:41:09.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.856+0000 7f95237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:09.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.856+0000 7f95237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.856+0000 7f95237fe640 10 monclient: dump: 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.856+0000 7f952af5d640 10 monclient: get_monmap_and_config success 2024-03-25T17:41:09.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.856+0000 7f952af5d640 4 set_mon_vals no callback set 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.857+0000 7f952af5d640 10 monclient: shutdown 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.857+0000 7f952af5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.857+0000 7f952af5d640 1 -- 172.21.15.44:0/142040941 >> v1:172.21.15.44:6789/0 conn(0x7f952414be90 legacy=0x7f952414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.857+0000 7f952af5d640 1 -- 172.21.15.44:0/142040941 shutdown_connections 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.857+0000 7f952af5d640 1 -- 172.21.15.44:0/142040941 >> 172.21.15.44:0/142040941 conn(0x7f95240617a0 msgr2=0x7f9524061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.857+0000 7f952af5d640 1 -- 172.21.15.44:0/142040941 shutdown_connections 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.857+0000 7f952af5d640 1 -- 172.21.15.44:0/142040941 wait complete. 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.858+0000 7f952af5d640 10 monclient: build_initial_monmap 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.858+0000 7f952af5d640 10 monclient: monmap: 2024-03-25T17:41:09.859 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.858+0000 7f952af5d640 1 librados: starting msgr at 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.858+0000 7f952af5d640 1 librados: starting objecter 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.858+0000 7f952af5d640 1 Processor -- start 2024-03-25T17:41:09.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.859+0000 7f952af5d640 1 -- start start 2024-03-25T17:41:09.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.859+0000 7f952af5d640 1 librados: setting wanted keys 2024-03-25T17:41:09.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.859+0000 7f952af5d640 1 librados: calling monclient init 2024-03-25T17:41:09.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.859+0000 7f952af5d640 10 monclient: init 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.859+0000 7f952af5d640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.859+0000 7f952af5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.859+0000 7f952af5d640 10 monclient(hunting): picked mon.noname-c con 0x7f95240b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.860+0000 7f952af5d640 10 monclient(hunting): picked mon.noname-b con 0x7f95240b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.860+0000 7f952af5d640 10 monclient(hunting): picked mon.noname-a con 0x7f952414be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.860+0000 7f952af5d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95241d7af0 con 0x7f952414be90 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.860+0000 7f952af5d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952413a580 con 0x7f95240b4d60 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.860+0000 7f952af5d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952413a860 con 0x7f95240b1730 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.860+0000 7f952af5d640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:09.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.860+0000 7f952af5d640 10 monclient(hunting): authenticate will time out at 6073.087898s 2024-03-25T17:41:09.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.869+0000 7f9523fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f95240b4d60 0x7f95241d7f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:38700/0 (socket says 172.21.15.44:38700) 2024-03-25T17:41:09.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.869+0000 7f9523fff640 1 -- 172.21.15.44:0/3077639316 learned_addr learned my addr 172.21.15.44:0/3077639316 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:09.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.870+0000 7f95217fa640 10 client.?.objecter ms_handle_connect 0x7f95240b4d60 2024-03-25T17:41:09.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.870+0000 7f95217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:09.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.873+0000 7f95217fa640 10 client.?.objecter ms_handle_connect 0x7f952414be90 2024-03-25T17:41:09.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.873+0000 7f95217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:09.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.875+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332152288 0 0) 0x7f952413a580 con 0x7f95240b4d60 2024-03-25T17:41:09.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.875+0000 7f95217fa640 10 monclient(hunting): _negotiate 2024-03-25T17:41:09.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.875+0000 7f95217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:09.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.875+0000 7f95217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:09.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.875+0000 7f95217fa640 10 monclient(hunting): my global_id is 4315 2024-03-25T17:41:09.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.875+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9504004eb0 con 0x7f95240b4d60 2024-03-25T17:41:09.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.879+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507791768 0 0) 0x7f95241d7af0 con 0x7f952414be90 2024-03-25T17:41:09.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.879+0000 7f95217fa640 10 monclient(hunting): _negotiate 2024-03-25T17:41:09.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.879+0000 7f95217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:09.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.879+0000 7f95217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:09.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.879+0000 7f95217fa640 10 monclient(hunting): my global_id is 4329 2024-03-25T17:41:09.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.879+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f952413a580 con 0x7f952414be90 2024-03-25T17:41:09.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.881+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1966198241 0 0) 0x7f9504004eb0 con 0x7f95240b4d60 2024-03-25T17:41:09.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.881+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95241d7af0 con 0x7f95240b4d60 2024-03-25T17:41:09.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.881+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f950c001950 con 0x7f95240b4d60 2024-03-25T17:41:09.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.881+0000 7f95217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:09.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.881+0000 7f95217fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.881+0000 7f95217fa640 10 monclient(hunting): dump: 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.881+0000 7f95217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.882+0000 7f95217fa640 10 client.?.objecter ms_handle_connect 0x7f95240b1730 2024-03-25T17:41:09.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.882+0000 7f95217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:09.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2711503986 0 0) 0x7f95241d7af0 con 0x7f95240b4d60 2024-03-25T17:41:09.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 >> v1:172.21.15.44:6790/0 conn(0x7f95240b1730 legacy=0x7f952413af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:09.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 >> v1:172.21.15.44:6789/0 conn(0x7f952414be90 legacy=0x7f95241370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:09.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 10 monclient: _finish_hunting 0 2024-03-25T17:41:09.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 1 monclient: found mon.b 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95241d7730 con 0x7f95240b4d60 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 10 monclient: _finish_auth 0 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 10 monclient: _check_auth_tickets 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.890+0000 7f95217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 5 monclient: authenticate success, global_id 4315 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 10 monclient: _renew_subs 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95241377d0 con 0x7f95240b4d60 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 10 monclient: _renew_subs 2024-03-25T17:41:09.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:09.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9524139140 con 0x7f95240b4d60 2024-03-25T17:41:09.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.891+0000 7f952af5d640 1 librados: init done 2024-03-25T17:41:09.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.892+0000 7f952af5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:09.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.892+0000 7f952af5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:09.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.892+0000 7f952af5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:09.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.892+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95241d7ca0 con 0x7f95240b4d60 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.910+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f950c0047e0 con 0x7f95240b4d60 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.910+0000 7f95217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.910+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f950c004950 con 0x7f95240b4d60 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.910+0000 7f95217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.910+0000 7f9522ffd640 4 set_mon_vals no callback set 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.910+0000 7f95217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.910+0000 7f95217fa640 10 monclient: dump: 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:09.912 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:09.913 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:09.913 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:09.913 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:09.913 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:09.913 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:09.913 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:09.913 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:09.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.911+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f950c01a960 con 0x7f95240b4d60 2024-03-25T17:41:09.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.912+0000 7f9528cf9640 10 monclient: get_auth_request con 0x7f950406dd40 auth_method 0 2024-03-25T17:41:09.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.912+0000 7f95217fa640 10 client.4315.objecter ms_handle_connect 0x7f950406dd40 2024-03-25T17:41:09.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.913+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f950c0546d0 con 0x7f95240b4d60 2024-03-25T17:41:09.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.913+0000 7f95217fa640 10 client.4315.objecter ms_dispatch 0x7f9524000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:41:09.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.913+0000 7f95217fa640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:41:09.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.913+0000 7f95217fa640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:41:09.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:09.913+0000 7f95217fa640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-25T17:41:10.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.174+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f950c050010 con 0x7f95240b4d60 2024-03-25T17:41:10.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.174+0000 7f95217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:10.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.174+0000 7f95217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:41:10.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.400+0000 7f952af5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:10.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.400+0000 7f952af5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:10.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.400+0000 7f952af5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:10.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.400+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95240627b0 con 0x7f95240b4d60 2024-03-25T17:41:10.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.594+0000 7f95217fa640 1 -- 172.21.15.44:0/3077639316 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f950c052110 con 0x7f95240b4d60 2024-03-25T17:41:10.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.594+0000 7f95217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:10.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.594+0000 7f95217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:41:10.596 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:41:10.596 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:41:10.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 10 librados: watch_flush enter 2024-03-25T17:41:10.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 10 librados: watch_flush exit 2024-03-25T17:41:10.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-25T17:41:10.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 10 client.4315.objecter successfully canceled tick 2024-03-25T17:41:10.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 >> v1:172.21.15.44:6800/3257043132 conn(0x7f950406dd40 legacy=0x7f9504070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:10.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 10 monclient: shutdown 2024-03-25T17:41:10.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:10.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 >> v1:172.21.15.111:6789/0 conn(0x7f95240b4d60 legacy=0x7f95241d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:10.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 shutdown_connections 2024-03-25T17:41:10.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 >> 172.21.15.44:0/3077639316 conn(0x7f95240617a0 msgr2=0x7f95240b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:10.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 shutdown_connections 2024-03-25T17:41:10.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 1 -- 172.21.15.44:0/3077639316 wait complete. 2024-03-25T17:41:10.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:10.597+0000 7f952af5d640 1 librados: shutdown 2024-03-25T17:41:10.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:41:17.410 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:34.765 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.417 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:41:17.426 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:34.781 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.464 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:34.784 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.468 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:34.788 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.478 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:34.834 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.483 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:34.839 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.489 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:34.845 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.497 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:34.817 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.524+0000 7f06a3332640 10 monclient: get_monmap_and_config 2024-03-25T17:41:17.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.524+0000 7f06a3332640 10 monclient: build_initial_monmap 2024-03-25T17:41:17.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.524+0000 7f06a3332640 10 monclient: monmap: 2024-03-25T17:41:17.524 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:17.524 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:17.525 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:41:17.525206+0000 2024-03-25T17:41:17.525 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:41:17.525206+0000 2024-03-25T17:41:17.525 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:17.525 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:17.525 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:17.525 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:41:17.525 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:41:17.525 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:17.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.524+0000 7f06a3332640 1 Processor -- start 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 1 -- start start 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 10 monclient: init 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 10 monclient(hunting): picked mon.noname-b con 0x7f069c0b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 10 monclient(hunting): picked mon.noname-c con 0x7f069c0b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 10 monclient(hunting): picked mon.noname-a con 0x7f069c14be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0af940 con 0x7f069c14be90 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0b0a70 con 0x7f069c0b1730 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0b0d50 con 0x7f069c0b4d60 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:17.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.525+0000 7f06a3332640 10 monclient(hunting): authenticate will time out at 6080.753503s 2024-03-25T17:41:17.528 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:34.883 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:17.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.559+0000 7f06a08cd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f069c0b1730 0x7f069c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:43404/0 (socket says 172.21.15.44:43404) 2024-03-25T17:41:17.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.559+0000 7f06a08cd640 1 -- 172.21.15.44:0/2014973524 learned_addr learned my addr 172.21.15.44:0/2014973524 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:17.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.575+0000 7f0693fff640 1 -- 172.21.15.44:0/2014973524 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722740999 0 0) 0x7f069c0b0a70 con 0x7f069c0b1730 2024-03-25T17:41:17.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.575+0000 7f0693fff640 10 monclient(hunting): _negotiate 2024-03-25T17:41:17.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.575+0000 7f0693fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:17.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.575+0000 7f0693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:17.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.575+0000 7f0693fff640 10 monclient(hunting): my global_id is 4318 2024-03-25T17:41:17.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.575+0000 7f0693fff640 1 -- 172.21.15.44:0/2014973524 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688003520 con 0x7f069c0b1730 2024-03-25T17:41:17.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 1 -- 172.21.15.44:0/2014973524 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034736854 0 0) 0x7f0688003520 con 0x7f069c0b1730 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 1 -- 172.21.15.44:0/2014973524 >> v1:172.21.15.44:6790/0 conn(0x7f069c0b4d60 legacy=0x7f069c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 1 -- 172.21.15.44:0/2014973524 >> v1:172.21.15.44:6789/0 conn(0x7f069c14be90 legacy=0x7f069c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 10 monclient: _finish_hunting 0 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 1 monclient: found mon.noname-c 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 1 -- 172.21.15.44:0/2014973524 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0b1030 con 0x7f069c0b1730 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 10 monclient: _finish_auth 0 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 10 monclient: _check_auth_tickets 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 1 -- 172.21.15.44:0/2014973524 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f068c001690 con 0x7f069c0b1730 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f06a3332640 5 monclient: authenticate success, global_id 4318 2024-03-25T17:41:17.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f06a3332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f0693fff640 10 monclient: dump: 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:17.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.593+0000 7f06a3332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:17.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.602+0000 7f0693fff640 1 -- 172.21.15.44:0/2014973524 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f068c001860 con 0x7f069c0b1730 2024-03-25T17:41:17.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.603+0000 7f0693fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:17.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.603+0000 7f06a3332640 10 monclient: get_monmap_and_config success 2024-03-25T17:41:17.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.603+0000 7f06a3332640 4 set_mon_vals no callback set 2024-03-25T17:41:17.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.603+0000 7f06a3332640 10 monclient: shutdown 2024-03-25T17:41:17.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.603+0000 7f06a3332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:17.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.603+0000 7f06a3332640 1 -- 172.21.15.44:0/2014973524 >> v1:172.21.15.111:6789/0 conn(0x7f069c0b1730 legacy=0x7f069c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:17.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.603+0000 7f06a3332640 1 -- 172.21.15.44:0/2014973524 shutdown_connections 2024-03-25T17:41:17.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.604+0000 7f06a3332640 1 -- 172.21.15.44:0/2014973524 >> 172.21.15.44:0/2014973524 conn(0x7f069c0617a0 msgr2=0x7f069c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:17.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.604+0000 7f06a3332640 1 -- 172.21.15.44:0/2014973524 shutdown_connections 2024-03-25T17:41:17.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.604+0000 7f06a3332640 1 -- 172.21.15.44:0/2014973524 wait complete. 2024-03-25T17:41:17.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.604+0000 7f06a3332640 10 monclient: build_initial_monmap 2024-03-25T17:41:17.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.604+0000 7f06a3332640 10 monclient: monmap: 2024-03-25T17:41:17.605 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.605+0000 7f06a3332640 1 librados: starting msgr at 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.605+0000 7f06a3332640 1 librados: starting objecter 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.605+0000 7f06a3332640 1 Processor -- start 2024-03-25T17:41:17.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.605+0000 7f06a3332640 1 -- start start 2024-03-25T17:41:17.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.605+0000 7f06a3332640 1 librados: setting wanted keys 2024-03-25T17:41:17.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.605+0000 7f06a3332640 1 librados: calling monclient init 2024-03-25T17:41:17.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.605+0000 7f06a3332640 10 monclient: init 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 10 monclient(hunting): picked mon.noname-b con 0x7f069c0b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 10 monclient(hunting): picked mon.noname-a con 0x7f069c0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 10 monclient(hunting): picked mon.noname-c con 0x7f069c14be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c142f60 con 0x7f069c0b4d60 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c135650 con 0x7f069c0b1730 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c1357e0 con 0x7f069c14be90 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:17.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.606+0000 7f06a3332640 10 monclient(hunting): authenticate will time out at 6080.834139s 2024-03-25T17:41:17.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.613+0000 7f06a10ce640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f069c0b1730 0x7f069c134f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:43422/0 (socket says 172.21.15.44:43422) 2024-03-25T17:41:17.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.613+0000 7f06a10ce640 1 -- 172.21.15.44:0/655564212 learned_addr learned my addr 172.21.15.44:0/655564212 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:17.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.616+0000 7f0691ffb640 10 client.?.objecter ms_handle_connect 0x7f069c0b1730 2024-03-25T17:41:17.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.616+0000 7f0691ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:17.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.616+0000 7f0691ffb640 10 client.?.objecter ms_handle_connect 0x7f069c0b4d60 2024-03-25T17:41:17.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.616+0000 7f0691ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:17.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.622+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779167227 0 0) 0x7f069c135650 con 0x7f069c0b1730 2024-03-25T17:41:17.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.622+0000 7f0691ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:41:17.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.622+0000 7f0691ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:17.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.622+0000 7f0691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:17.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.622+0000 7f0691ffb640 10 monclient(hunting): my global_id is 4321 2024-03-25T17:41:17.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.622+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0680004eb0 con 0x7f069c0b1730 2024-03-25T17:41:17.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.624+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847116306 0 0) 0x7f069c142f60 con 0x7f069c0b4d60 2024-03-25T17:41:17.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.624+0000 7f0691ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:41:17.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.624+0000 7f0691ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:17.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.624+0000 7f0691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:17.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.624+0000 7f0691ffb640 10 monclient(hunting): my global_id is 4335 2024-03-25T17:41:17.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.624+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c135650 con 0x7f069c0b4d60 2024-03-25T17:41:17.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.634+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1752134669 0 0) 0x7f0680004eb0 con 0x7f069c0b1730 2024-03-25T17:41:17.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.634+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f069c142f60 con 0x7f069c0b1730 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.634+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0698004080 con 0x7f069c0b1730 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.634+0000 7f0691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.634+0000 7f0691ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.634+0000 7f0691ffb640 10 monclient(hunting): dump: 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:17.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.634+0000 7f0691ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:41:17.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.635+0000 7f0691ffb640 10 client.?.objecter ms_handle_connect 0x7f069c14be90 2024-03-25T17:41:17.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.635+0000 7f0691ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.641+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 135715571 0 0) 0x7f069c135650 con 0x7f069c0b4d60 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0680004eb0 con 0x7f069c0b4d60 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f068c001bb0 con 0x7f069c0b4d60 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 10 monclient(hunting): dump: 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:17.642 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:17.643 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:17.643 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:17.643 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:17.643 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:17.643 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:17.643 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:17.643 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:17.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4127576113 0 0) 0x7f069c142f60 con 0x7f069c0b1730 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 >> v1:172.21.15.44:6790/0 conn(0x7f069c14be90 legacy=0x7f069c1d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 >> v1:172.21.15.44:6789/0 conn(0x7f069c0b4d60 legacy=0x7f069c1d7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 1 monclient: found mon.b 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c139440 con 0x7f069c0b1730 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 10 monclient: _finish_auth 0 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f0691ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f06a3332640 5 monclient: authenticate success, global_id 4321 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f06a3332640 10 monclient: _renew_subs 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f06a3332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:17.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.642+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c139690 con 0x7f069c0b1730 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.643+0000 7f06a3332640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.643+0000 7f06a3332640 10 monclient: _renew_subs 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.643+0000 7f06a3332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.643+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c1390d0 con 0x7f069c0b1730 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.643+0000 7f06a3332640 1 librados: init done 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.644+0000 7f06a3332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.644+0000 7f06a3332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.644+0000 7f06a3332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:17.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.644+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c142dc0 con 0x7f069c0b1730 2024-03-25T17:41:17.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.867+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0698000ed0 con 0x7f069c0b1730 2024-03-25T17:41:17.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.867+0000 7f0691ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:17.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.867+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0698001100 con 0x7f069c0b1730 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.867+0000 7f0691ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.867+0000 7f0691ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.867+0000 7f0691ffb640 10 monclient: dump: 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:17.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.867+0000 7f06937fe640 4 set_mon_vals no callback set 2024-03-25T17:41:17.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.917+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f0698001260 con 0x7f069c0b1730 2024-03-25T17:41:17.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.917+0000 7f06a08cd640 10 monclient: get_auth_request con 0x7f068006dfe0 auth_method 0 2024-03-25T17:41:17.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.918+0000 7f0691ffb640 10 client.4321.objecter ms_handle_connect 0x7f068006dfe0 2024-03-25T17:41:17.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.920+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f0698054630 con 0x7f069c0b1730 2024-03-25T17:41:17.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.920+0000 7f0691ffb640 10 client.4321.objecter ms_dispatch 0x7f069c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:41:17.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.920+0000 7f0691ffb640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:41:17.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.920+0000 7f0691ffb640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:41:17.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.920+0000 7f0691ffb640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T17:41:17.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.926+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0698050140 con 0x7f069c0b1730 2024-03-25T17:41:17.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.926+0000 7f0691ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:17.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:17.926+0000 7f0691ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:41:18.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.174+0000 7f06a3332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:18.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.174+0000 7f06a3332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:18.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.174+0000 7f06a3332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:18.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.174+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f069c1431a0 con 0x7f069c0b1730 2024-03-25T17:41:18.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.201+0000 7f0691ffb640 1 -- 172.21.15.44:0/655564212 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f069c1431a0 con 0x7f069c0b1730 2024-03-25T17:41:18.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.201+0000 7f0691ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:18.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.201+0000 7f0691ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:41:18.202 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:41:18.202 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:41:18.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.203+0000 7f06a3332640 10 librados: watch_flush enter 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.203+0000 7f06a3332640 10 librados: watch_flush exit 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.203+0000 7f06a3332640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.203+0000 7f06a3332640 10 client.4321.objecter successfully canceled tick 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 >> v1:172.21.15.44:6800/3257043132 conn(0x7f068006dfe0 legacy=0x7f06800704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 10 monclient: shutdown 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 >> v1:172.21.15.111:6789/0 conn(0x7f069c0b1730 legacy=0x7f069c134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 shutdown_connections 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 >> 172.21.15.44:0/655564212 conn(0x7f069c0617a0 msgr2=0x7f069c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 shutdown_connections 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 1 -- 172.21.15.44:0/655564212 wait complete. 2024-03-25T17:41:18.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:18.204+0000 7f06a3332640 1 librados: shutdown 2024-03-25T17:41:18.216 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:41:25.024 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.133+0000 7fe4fad24640 10 monclient: get_monmap_and_config 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.133+0000 7fe4fad24640 10 monclient: build_initial_monmap 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.133+0000 7fe4fad24640 10 monclient: monmap: 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:41:25.134596+0000 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:41:25.134596+0000 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:25.134 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:25.135 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:25.135 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:41:25.135 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:41:25.135 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:25.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.134+0000 7fe4fad24640 1 Processor -- start 2024-03-25T17:41:25.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.134+0000 7fe4fad24640 1 -- start start 2024-03-25T17:41:25.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.134+0000 7fe4fad24640 10 monclient: init 2024-03-25T17:41:25.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:25.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f40b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f414be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40af940 con 0x7fe4f414be90 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40b0a70 con 0x7fe4f40b4d60 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40b0d50 con 0x7fe4f40b1730 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4fad24640 10 monclient(hunting): authenticate will time out at 6088.362814s 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4f8ac0640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe4f40b4d60 0x7fe4f413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45420/0 (socket says 172.21.15.44:45420) 2024-03-25T17:41:25.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.135+0000 7fe4f8ac0640 1 -- 172.21.15.44:0/941328326 learned_addr learned my addr 172.21.15.44:0/941328326 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:25.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.163+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458422306 0 0) 0x7fe4f40b0d50 con 0x7fe4f40b1730 2024-03-25T17:41:25.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.163+0000 7fe4f37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:25.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.163+0000 7fe4f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:25.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.163+0000 7fe4f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:25.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.163+0000 7fe4f37fe640 10 monclient(hunting): my global_id is 4322 2024-03-25T17:41:25.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.163+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4003520 con 0x7fe4f40b1730 2024-03-25T17:41:25.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.170+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030630235 0 0) 0x7fe4f40b0a70 con 0x7fe4f40b4d60 2024-03-25T17:41:25.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.170+0000 7fe4f37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:25.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.170+0000 7fe4f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:25.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.170+0000 7fe4f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:25.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.170+0000 7fe4f37fe640 10 monclient(hunting): my global_id is 4324 2024-03-25T17:41:25.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.170+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f40b0d50 con 0x7fe4f40b4d60 2024-03-25T17:41:25.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2723014199 0 0) 0x7fe4e4003520 con 0x7fe4f40b1730 2024-03-25T17:41:25.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 >> v1:172.21.15.111:6789/0 conn(0x7fe4f40b4d60 legacy=0x7fe4f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:25.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 >> v1:172.21.15.44:6789/0 conn(0x7fe4f414be90 legacy=0x7fe4f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:41:25.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:41:25.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 1 monclient: found mon.noname-b 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f40b1030 con 0x7fe4f40b1730 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 10 monclient: _finish_auth 0 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe4e0001690 con 0x7fe4f40b1730 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.172+0000 7fe4f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.173+0000 7fe4f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.173+0000 7fe4f37fe640 10 monclient: dump: 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:25.174 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.173+0000 7fe4fad24640 5 monclient: authenticate success, global_id 4322 2024-03-25T17:41:25.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.173+0000 7fe4fad24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:25.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4e00017f0 con 0x7fe4f40b1730 2024-03-25T17:41:25.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:25.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4f37fe640 1 -- 172.21.15.44:0/941328326 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe4e0001c00 con 0x7fe4f40b1730 2024-03-25T17:41:25.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4f37fe640 10 monclient: dump: 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4fad24640 10 monclient: get_monmap_and_config success 2024-03-25T17:41:25.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4fad24640 4 set_mon_vals no callback set 2024-03-25T17:41:25.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4fad24640 10 monclient: shutdown 2024-03-25T17:41:25.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4fad24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:25.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.196+0000 7fe4fad24640 1 -- 172.21.15.44:0/941328326 >> v1:172.21.15.44:6790/0 conn(0x7fe4f40b1730 legacy=0x7fe4f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:25.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.197+0000 7fe4fad24640 1 -- 172.21.15.44:0/941328326 shutdown_connections 2024-03-25T17:41:25.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.197+0000 7fe4fad24640 1 -- 172.21.15.44:0/941328326 >> 172.21.15.44:0/941328326 conn(0x7fe4f40617a0 msgr2=0x7fe4f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:25.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.197+0000 7fe4fad24640 1 -- 172.21.15.44:0/941328326 shutdown_connections 2024-03-25T17:41:25.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.197+0000 7fe4fad24640 1 -- 172.21.15.44:0/941328326 wait complete. 2024-03-25T17:41:25.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.198+0000 7fe4fad24640 10 monclient: build_initial_monmap 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.198+0000 7fe4fad24640 10 monclient: monmap: 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.198+0000 7fe4fad24640 1 librados: starting msgr at 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.198+0000 7fe4fad24640 1 librados: starting objecter 2024-03-25T17:41:25.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.198+0000 7fe4fad24640 1 Processor -- start 2024-03-25T17:41:25.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.199+0000 7fe4fad24640 1 -- start start 2024-03-25T17:41:25.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.199+0000 7fe4fad24640 1 librados: setting wanted keys 2024-03-25T17:41:25.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.199+0000 7fe4fad24640 1 librados: calling monclient init 2024-03-25T17:41:25.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.199+0000 7fe4fad24640 10 monclient: init 2024-03-25T17:41:25.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f40b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f40b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f414be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f413b020 con 0x7fe4f40b1730 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f413ab90 con 0x7fe4f414be90 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f413b970 con 0x7fe4f40b4d60 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:25.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.200+0000 7fe4fad24640 10 monclient(hunting): authenticate will time out at 6088.427808s 2024-03-25T17:41:25.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.212+0000 7fe4f92c1640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe4f414be90 0x7fe4f41368a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45482/0 (socket says 172.21.15.44:45482) 2024-03-25T17:41:25.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.212+0000 7fe4f92c1640 1 -- 172.21.15.44:0/3226568531 learned_addr learned my addr 172.21.15.44:0/3226568531 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:25.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.215+0000 7fe4f17fa640 10 client.?.objecter ms_handle_connect 0x7fe4f414be90 2024-03-25T17:41:25.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.215+0000 7fe4f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:25.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.219+0000 7fe4f17fa640 10 client.?.objecter ms_handle_connect 0x7fe4f40b4d60 2024-03-25T17:41:25.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.219+0000 7fe4f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:25.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.227+0000 7fe4f17fa640 10 client.?.objecter ms_handle_connect 0x7fe4f40b1730 2024-03-25T17:41:25.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.227+0000 7fe4f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:25.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.227+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431088879 0 0) 0x7fe4f413ab90 con 0x7fe4f414be90 2024-03-25T17:41:25.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.227+0000 7fe4f17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:41:25.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.227+0000 7fe4f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:25.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.227+0000 7fe4f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:25.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.227+0000 7fe4f17fa640 10 monclient(hunting): my global_id is 4327 2024-03-25T17:41:25.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.227+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8004eb0 con 0x7fe4f414be90 2024-03-25T17:41:25.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.238+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480918071 0 0) 0x7fe4f413b970 con 0x7fe4f40b4d60 2024-03-25T17:41:25.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.238+0000 7fe4f17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:41:25.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.238+0000 7fe4f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:25.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.238+0000 7fe4f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:25.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.238+0000 7fe4f17fa640 10 monclient(hunting): my global_id is 4325 2024-03-25T17:41:25.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.238+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f413ab90 con 0x7fe4f40b4d60 2024-03-25T17:41:25.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.240+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1188587334 0 0) 0x7fe4d8004eb0 con 0x7fe4f414be90 2024-03-25T17:41:25.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.240+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4f413b970 con 0x7fe4f414be90 2024-03-25T17:41:25.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.244+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166756999 0 0) 0x7fe4f413b020 con 0x7fe4f40b1730 2024-03-25T17:41:25.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.244+0000 7fe4f17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:41:25.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.244+0000 7fe4f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:25.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.244+0000 7fe4f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:25.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.244+0000 7fe4f17fa640 10 monclient(hunting): my global_id is 4338 2024-03-25T17:41:25.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.244+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8004eb0 con 0x7fe4f40b1730 2024-03-25T17:41:25.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.246+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe4ec001b90 con 0x7fe4f414be90 2024-03-25T17:41:25.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.246+0000 7fe4f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:25.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.246+0000 7fe4f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:25.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.246+0000 7fe4f17fa640 10 monclient(hunting): dump: 2024-03-25T17:41:25.246 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:25.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.246+0000 7fe4f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:41:25.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3785899173 0 0) 0x7fe4f413b970 con 0x7fe4f414be90 2024-03-25T17:41:25.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 >> v1:172.21.15.44:6790/0 conn(0x7fe4f40b4d60 legacy=0x7fe4f4136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:25.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 >> v1:172.21.15.44:6789/0 conn(0x7fe4f40b1730 legacy=0x7fe4f41418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:25.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:41:25.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 1 monclient: found mon.b 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4136fb0 con 0x7fe4f414be90 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 10 monclient: _finish_auth 0 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 5 monclient: authenticate success, global_id 4327 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 10 monclient: _renew_subs 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f413bbe0 con 0x7fe4f414be90 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 10 monclient: _renew_subs 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f4139f90 con 0x7fe4f414be90 2024-03-25T17:41:25.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.255+0000 7fe4fad24640 1 librados: init done 2024-03-25T17:41:25.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.256+0000 7fe4fad24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:25.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.256+0000 7fe4fad24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:25.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.256+0000 7fe4fad24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:25.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.256+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f406e150 con 0x7fe4f414be90 2024-03-25T17:41:25.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.531+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4ec002020 con 0x7fe4f414be90 2024-03-25T17:41:25.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.531+0000 7fe4f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:25.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.531+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe4ec004d80 con 0x7fe4f414be90 2024-03-25T17:41:25.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.531+0000 7fe4f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.531+0000 7fe4f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.531+0000 7fe4f17fa640 10 monclient: dump: 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:25.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.531+0000 7fe4f2ffd640 4 set_mon_vals no callback set 2024-03-25T17:41:25.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.534+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fe4ec01ad50 con 0x7fe4f414be90 2024-03-25T17:41:25.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.534+0000 7fe4f8ac0640 10 monclient: get_auth_request con 0x7fe4d806dd60 auth_method 0 2024-03-25T17:41:25.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.534+0000 7fe4f17fa640 10 client.4327.objecter ms_handle_connect 0x7fe4d806dd60 2024-03-25T17:41:25.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.538+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fe4ec054a40 con 0x7fe4f414be90 2024-03-25T17:41:25.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.538+0000 7fe4f17fa640 10 client.4327.objecter ms_dispatch 0x7fe4f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:41:25.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.538+0000 7fe4f17fa640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:41:25.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.538+0000 7fe4f17fa640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:41:25.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.538+0000 7fe4f17fa640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-25T17:41:25.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.539+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4ec050380 con 0x7fe4f414be90 2024-03-25T17:41:25.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.539+0000 7fe4f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:25.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.539+0000 7fe4f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:41:25.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.788+0000 7fe4fad24640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:25.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.788+0000 7fe4fad24640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:25.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.788+0000 7fe4fad24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:25.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.788+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4f40627b0 con 0x7fe4f414be90 2024-03-25T17:41:25.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.807+0000 7fe4f17fa640 1 -- 172.21.15.44:0/3226568531 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fe4ec052480 con 0x7fe4f414be90 2024-03-25T17:41:25.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.807+0000 7fe4f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:25.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.807+0000 7fe4f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:41:25.808 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:41:25.808 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 10 librados: watch_flush enter 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 10 librados: watch_flush exit 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 10 client.4327.objecter successfully canceled tick 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 >> v1:172.21.15.44:6800/3257043132 conn(0x7fe4d806dd60 legacy=0x7fe4d8070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 10 monclient: shutdown 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 >> v1:172.21.15.111:6789/0 conn(0x7fe4f414be90 legacy=0x7fe4f41368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 shutdown_connections 2024-03-25T17:41:25.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 >> 172.21.15.44:0/3226568531 conn(0x7fe4f40617a0 msgr2=0x7fe4f413c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:25.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 shutdown_connections 2024-03-25T17:41:25.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 1 -- 172.21.15.44:0/3226568531 wait complete. 2024-03-25T17:41:25.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:25.810+0000 7fe4fad24640 1 librados: shutdown 2024-03-25T17:41:25.822 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:41:32.629 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:41:32.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.746+0000 7f5189113640 10 monclient: get_monmap_and_config 2024-03-25T17:41:32.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.746+0000 7f5189113640 10 monclient: build_initial_monmap 2024-03-25T17:41:32.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.746+0000 7f5189113640 10 monclient: monmap: 2024-03-25T17:41:32.747 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:32.747 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:32.747 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:41:32.747762+0000 2024-03-25T17:41:32.747 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:41:32.747762+0000 2024-03-25T17:41:32.747 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:32.748 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:32.748 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:32.748 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:41:32.748 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:41:32.748 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:32.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.747+0000 7f5189113640 1 Processor -- start 2024-03-25T17:41:32.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.747+0000 7f5189113640 1 -- start start 2024-03-25T17:41:32.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.747+0000 7f5189113640 10 monclient: init 2024-03-25T17:41:32.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:32.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:41:32.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 10 monclient(hunting): picked mon.noname-b con 0x7f51840b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:32.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 10 monclient(hunting): picked mon.noname-c con 0x7f51840b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:32.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 10 monclient(hunting): picked mon.noname-a con 0x7f518414be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:32.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51840af940 con 0x7f518414be90 2024-03-25T17:41:32.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51840b0a70 con 0x7f51840b1730 2024-03-25T17:41:32.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51840b0d50 con 0x7f51840b4d60 2024-03-25T17:41:32.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:32.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.748+0000 7f5189113640 10 monclient(hunting): authenticate will time out at 6095.976057s 2024-03-25T17:41:32.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.758+0000 7f518259c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f51840b1730 0x7f51840b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:42008/0 (socket says 172.21.15.44:42008) 2024-03-25T17:41:32.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.758+0000 7f518259c640 1 -- 172.21.15.44:0/2340458661 learned_addr learned my addr 172.21.15.44:0/2340458661 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:32.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.763+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250908138 0 0) 0x7f51840b0a70 con 0x7f51840b1730 2024-03-25T17:41:32.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.763+0000 7f5181d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:41:32.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.763+0000 7f5181d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:32.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.763+0000 7f5181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:32.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.763+0000 7f5181d9b640 10 monclient(hunting): my global_id is 4330 2024-03-25T17:41:32.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.763+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f516c003520 con 0x7f51840b1730 2024-03-25T17:41:32.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.776+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024625898 0 0) 0x7f51840af940 con 0x7f518414be90 2024-03-25T17:41:32.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.776+0000 7f5181d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:41:32.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.776+0000 7f5181d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:32.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.776+0000 7f5181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:32.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.776+0000 7f5181d9b640 10 monclient(hunting): my global_id is 4341 2024-03-25T17:41:32.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.776+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51840b0a70 con 0x7f518414be90 2024-03-25T17:41:32.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.781+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752341854 0 0) 0x7f516c003520 con 0x7f51840b1730 2024-03-25T17:41:32.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 >> v1:172.21.15.44:6790/0 conn(0x7f51840b4d60 legacy=0x7f518413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:41:32.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 >> v1:172.21.15.44:6789/0 conn(0x7f518414be90 legacy=0x7f518414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:32.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 1 monclient: found mon.noname-c 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51840b1030 con 0x7f51840b1730 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 10 monclient: _finish_auth 0 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5178001690 con 0x7f51840b1730 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5181d9b640 10 monclient: dump: 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:32.783 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5189113640 5 monclient: authenticate success, global_id 4330 2024-03-25T17:41:32.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.782+0000 7f5189113640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:32.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51780017f0 con 0x7f51840b1730 2024-03-25T17:41:32.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5181d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:32.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5181d9b640 1 -- 172.21.15.44:0/2340458661 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5178001c00 con 0x7f51840b1730 2024-03-25T17:41:32.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5181d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:32.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5181d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:41:32.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5181d9b640 10 monclient: dump: 2024-03-25T17:41:32.789 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 10 monclient: get_monmap_and_config success 2024-03-25T17:41:32.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 4 set_mon_vals no callback set 2024-03-25T17:41:32.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 10 monclient: shutdown 2024-03-25T17:41:32.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:32.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 1 -- 172.21.15.44:0/2340458661 >> v1:172.21.15.111:6789/0 conn(0x7f51840b1730 legacy=0x7f51840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:32.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 1 -- 172.21.15.44:0/2340458661 shutdown_connections 2024-03-25T17:41:32.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 1 -- 172.21.15.44:0/2340458661 >> 172.21.15.44:0/2340458661 conn(0x7f51840617a0 msgr2=0x7f5184061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:32.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 1 -- 172.21.15.44:0/2340458661 shutdown_connections 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.789+0000 7f5189113640 1 -- 172.21.15.44:0/2340458661 wait complete. 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.790+0000 7f5189113640 10 monclient: build_initial_monmap 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.790+0000 7f5189113640 10 monclient: monmap: 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:32.794 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.790+0000 7f5189113640 1 librados: starting msgr at 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.790+0000 7f5189113640 1 librados: starting objecter 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.791+0000 7f5189113640 1 Processor -- start 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.791+0000 7f5189113640 1 -- start start 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.791+0000 7f5189113640 1 librados: setting wanted keys 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.791+0000 7f5189113640 1 librados: calling monclient init 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.791+0000 7f5189113640 10 monclient: init 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.791+0000 7f5189113640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.791+0000 7f5189113640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:41:32.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.791+0000 7f5189113640 10 monclient(hunting): picked mon.noname-c con 0x7f51840b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:32.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.792+0000 7f5189113640 10 monclient(hunting): picked mon.noname-b con 0x7f51840b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:32.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.792+0000 7f5189113640 10 monclient(hunting): picked mon.noname-a con 0x7f518414be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:32.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.792+0000 7f5189113640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5184138840 con 0x7f518414be90 2024-03-25T17:41:32.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.792+0000 7f5189113640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51841d8620 con 0x7f51840b4d60 2024-03-25T17:41:32.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.792+0000 7f5189113640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51841d79f0 con 0x7f51840b1730 2024-03-25T17:41:32.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.792+0000 7f5189113640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:32.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.792+0000 7f5189113640 10 monclient(hunting): authenticate will time out at 6096.019865s 2024-03-25T17:41:32.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.796+0000 7f5182d9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f51840b1730 0x7f51841d7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:48090/0 (socket says 172.21.15.44:48090) 2024-03-25T17:41:32.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.796+0000 7f5182d9d640 1 -- 172.21.15.44:0/2859334202 learned_addr learned my addr 172.21.15.44:0/2859334202 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:32.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.796+0000 7f516b7fe640 10 client.?.objecter ms_handle_connect 0x7f51840b1730 2024-03-25T17:41:32.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.796+0000 7f516b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:32.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.807+0000 7f516b7fe640 10 client.?.objecter ms_handle_connect 0x7f51840b4d60 2024-03-25T17:41:32.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.807+0000 7f516b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:32.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.810+0000 7f516b7fe640 10 client.?.objecter ms_handle_connect 0x7f518414be90 2024-03-25T17:41:32.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.810+0000 7f516b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:32.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.817+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200852727 0 0) 0x7f5184138840 con 0x7f518414be90 2024-03-25T17:41:32.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.817+0000 7f516b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:32.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.817+0000 7f516b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:32.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.817+0000 7f516b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:32.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.817+0000 7f516b7fe640 10 monclient(hunting): my global_id is 4344 2024-03-25T17:41:32.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.817+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5170004eb0 con 0x7f518414be90 2024-03-25T17:41:32.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.823+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988577244 0 0) 0x7f51841d8620 con 0x7f51840b4d60 2024-03-25T17:41:32.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.823+0000 7f516b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:32.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.823+0000 7f516b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:32.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.823+0000 7f516b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:32.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.823+0000 7f516b7fe640 10 monclient(hunting): my global_id is 4333 2024-03-25T17:41:32.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.823+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5184138840 con 0x7f51840b4d60 2024-03-25T17:41:32.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.825+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234208893 0 0) 0x7f51841d79f0 con 0x7f51840b1730 2024-03-25T17:41:32.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.825+0000 7f516b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:32.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.825+0000 7f516b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:32.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.825+0000 7f516b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:32.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.825+0000 7f516b7fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T17:41:32.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.825+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51841d8620 con 0x7f51840b1730 2024-03-25T17:41:32.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.831+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 483614703 0 0) 0x7f5184138840 con 0x7f51840b4d60 2024-03-25T17:41:32.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.831+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51841d79f0 con 0x7f51840b4d60 2024-03-25T17:41:32.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.831+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5178001b00 con 0x7f51840b4d60 2024-03-25T17:41:32.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.831+0000 7f516b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.831+0000 7f516b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.831+0000 7f516b7fe640 10 monclient(hunting): dump: 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:32.833 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:32.834 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:32.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.831+0000 7f516b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:41:32.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.838+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2235211611 0 0) 0x7f5170004eb0 con 0x7f518414be90 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.838+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5184138840 con 0x7f518414be90 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.838+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f517c001bf0 con 0x7f518414be90 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.838+0000 7f516b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.838+0000 7f516b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.838+0000 7f516b7fe640 10 monclient(hunting): dump: 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:32.839 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:32.840 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:32.840 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:32.840 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:32.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1462654155 0 0) 0x7f51841d79f0 con 0x7f51840b4d60 2024-03-25T17:41:32.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 >> v1:172.21.15.44:6790/0 conn(0x7f51840b1730 legacy=0x7f51841d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:32.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 >> v1:172.21.15.44:6789/0 conn(0x7f518414be90 legacy=0x7f5184138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:32.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 1 monclient: found mon.b 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5184142f20 con 0x7f51840b4d60 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f516b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f5189113640 5 monclient: authenticate success, global_id 4333 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f5189113640 10 monclient: _renew_subs 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f5189113640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.849+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5184143170 con 0x7f51840b4d60 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.850+0000 7f5189113640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:41:32.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.850+0000 7f5189113640 10 monclient: _renew_subs 2024-03-25T17:41:32.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.850+0000 7f5189113640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:32.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.850+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5184140560 con 0x7f51840b4d60 2024-03-25T17:41:32.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.850+0000 7f5189113640 1 librados: init done 2024-03-25T17:41:32.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.851+0000 7f5189113640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:32.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.851+0000 7f5189113640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:32.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.851+0000 7f5189113640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:32.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.851+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51841d7860 con 0x7f51840b4d60 2024-03-25T17:41:32.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.869+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5178001ca0 con 0x7f51840b4d60 2024-03-25T17:41:32.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.869+0000 7f516b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:32.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.869+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5178004c30 con 0x7f51840b4d60 2024-03-25T17:41:32.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.869+0000 7f516b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:32.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.870+0000 7f516b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:41:32.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.869+0000 7f5180d99640 4 set_mon_vals no callback set 2024-03-25T17:41:32.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.870+0000 7f516b7fe640 10 monclient: dump: 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:32.871 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:32.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.870+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f517801ac00 con 0x7f51840b4d60 2024-03-25T17:41:32.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.871+0000 7f5182d9d640 10 monclient: get_auth_request con 0x7f517006df70 auth_method 0 2024-03-25T17:41:32.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.871+0000 7f516b7fe640 10 client.4333.objecter ms_handle_connect 0x7f517006df70 2024-03-25T17:41:32.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.874+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f51780547c0 con 0x7f51840b4d60 2024-03-25T17:41:32.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.874+0000 7f516b7fe640 10 client.4333.objecter ms_dispatch 0x7f5184000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:41:32.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.874+0000 7f516b7fe640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:41:32.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.874+0000 7f516b7fe640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:41:32.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:32.874+0000 7f516b7fe640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T17:41:33.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.141+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51780502d0 con 0x7f51840b4d60 2024-03-25T17:41:33.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.142+0000 7f516b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:33.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.142+0000 7f516b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:41:33.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.400+0000 7f5189113640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:33.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.400+0000 7f5189113640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:33.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.400+0000 7f5189113640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:33.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.400+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5184062100 con 0x7f51840b4d60 2024-03-25T17:41:33.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.432+0000 7f516b7fe640 1 -- 172.21.15.44:0/2859334202 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f5178052200 con 0x7f51840b4d60 2024-03-25T17:41:33.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.432+0000 7f516b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:33.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.432+0000 7f516b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:41:33.432 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:41:33.433 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:41:33.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 10 librados: watch_flush enter 2024-03-25T17:41:33.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 10 librados: watch_flush exit 2024-03-25T17:41:33.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T17:41:33.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 10 client.4333.objecter successfully canceled tick 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 >> v1:172.21.15.44:6800/3257043132 conn(0x7f517006df70 legacy=0x7f5170070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 10 monclient: shutdown 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 >> v1:172.21.15.111:6789/0 conn(0x7f51840b4d60 legacy=0x7f5184137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 shutdown_connections 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.434+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 >> 172.21.15.44:0/2859334202 conn(0x7f51840617a0 msgr2=0x7f51840b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.435+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 shutdown_connections 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.435+0000 7f5189113640 1 -- 172.21.15.44:0/2859334202 wait complete. 2024-03-25T17:41:33.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:33.435+0000 7f5189113640 1 librados: shutdown 2024-03-25T17:41:33.448 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:41:37.509 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:54.865 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:37.522 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:54.878 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:37.557 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:54.913 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:37.570 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:54.890 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:37.572 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:54.893 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:37.594 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:54.914 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:37.600 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:54.956 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:37.606 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:54.962 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:37.643 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:54.999 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:40.254 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:41:40.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.367+0000 7fd7b16f1640 10 monclient: get_monmap_and_config 2024-03-25T17:41:40.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.367+0000 7fd7b16f1640 10 monclient: build_initial_monmap 2024-03-25T17:41:40.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.367+0000 7fd7b16f1640 10 monclient: monmap: 2024-03-25T17:41:40.368 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:40.368 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:40.368 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:41:40.368454+0000 2024-03-25T17:41:40.368 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:41:40.368454+0000 2024-03-25T17:41:40.368 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:40.369 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:40.369 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:40.369 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:41:40.369 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:41:40.369 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:40.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 1 Processor -- start 2024-03-25T17:41:40.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 1 -- start start 2024-03-25T17:41:40.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 10 monclient: init 2024-03-25T17:41:40.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:40.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:41:40.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:40.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:40.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:40.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac142520 con 0x7fd7ac0a98f0 2024-03-25T17:41:40.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac0b28a0 con 0x7fd7ac1430c0 2024-03-25T17:41:40.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac0b2a30 con 0x7fd7ac13f710 2024-03-25T17:41:40.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:40.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.368+0000 7fd7b16f1640 10 monclient(hunting): authenticate will time out at 6103.596652s 2024-03-25T17:41:40.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.373+0000 7fd7a3fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fd7ac13f710 0x7fd7ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:39260/0 (socket says 172.21.15.44:39260) 2024-03-25T17:41:40.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.373+0000 7fd7a3fff640 1 -- 172.21.15.44:0/1668467783 learned_addr learned my addr 172.21.15.44:0/1668467783 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:40.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.392+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277198452 0 0) 0x7fd7ac142520 con 0x7fd7ac0a98f0 2024-03-25T17:41:40.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.392+0000 7fd7aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:41:40.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.392+0000 7fd7aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:40.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.392+0000 7fd7aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:40.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.392+0000 7fd7aa7fc640 10 monclient(hunting): my global_id is 4347 2024-03-25T17:41:40.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.392+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd798003520 con 0x7fd7ac0a98f0 2024-03-25T17:41:40.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.400+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272172598 0 0) 0x7fd7ac0b2a30 con 0x7fd7ac13f710 2024-03-25T17:41:40.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.400+0000 7fd7aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:41:40.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.400+0000 7fd7aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:40.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.400+0000 7fd7aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:40.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.400+0000 7fd7aa7fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T17:41:40.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.400+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7ac142520 con 0x7fd7ac13f710 2024-03-25T17:41:40.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 226296778 0 0) 0x7fd7ac142520 con 0x7fd7ac13f710 2024-03-25T17:41:40.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 >> v1:172.21.15.111:6789/0 conn(0x7fd7ac1430c0 legacy=0x7fd7ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:40.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 >> v1:172.21.15.44:6789/0 conn(0x7fd7ac0a98f0 legacy=0x7fd7ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:40.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:41:40.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 1 monclient: found mon.noname-b 2024-03-25T17:41:40.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:40.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:41:40.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac0b44c0 con 0x7fd7ac13f710 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 10 monclient: _finish_auth 0 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd794001670 con 0x7fd7ac13f710 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7b16f1640 5 monclient: authenticate success, global_id 4331 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7b16f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7aa7fc640 10 monclient: dump: 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:40.408 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:40.409 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:40.409 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:40.409 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:40.409 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:40.409 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:40.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.406+0000 7fd7b16f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:40.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.424+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7940017d0 con 0x7fd7ac13f710 2024-03-25T17:41:40.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.424+0000 7fd7aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:40.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.424+0000 7fd7aa7fc640 1 -- 172.21.15.44:0/1668467783 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd794001ba0 con 0x7fd7ac13f710 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.424+0000 7fd7aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.424+0000 7fd7aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.424+0000 7fd7aa7fc640 10 monclient: dump: 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:40.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.425+0000 7fd7b16f1640 10 monclient: get_monmap_and_config success 2024-03-25T17:41:40.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.425+0000 7fd7b16f1640 4 set_mon_vals no callback set 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.425+0000 7fd7b16f1640 10 monclient: shutdown 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.425+0000 7fd7b16f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.425+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1668467783 >> v1:172.21.15.44:6790/0 conn(0x7fd7ac13f710 legacy=0x7fd7ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.425+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1668467783 shutdown_connections 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.425+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1668467783 >> 172.21.15.44:0/1668467783 conn(0x7fd7ac0b7ea0 msgr2=0x7fd7ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.426+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1668467783 shutdown_connections 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.426+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1668467783 wait complete. 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.426+0000 7fd7b16f1640 10 monclient: build_initial_monmap 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.427+0000 7fd7b16f1640 10 monclient: monmap: 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:40.429 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.427+0000 7fd7b16f1640 1 librados: starting msgr at 2024-03-25T17:41:40.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.427+0000 7fd7b16f1640 1 librados: starting objecter 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.427+0000 7fd7b16f1640 1 Processor -- start 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.427+0000 7fd7b16f1640 1 -- start start 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.427+0000 7fd7b16f1640 1 librados: setting wanted keys 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.427+0000 7fd7b16f1640 1 librados: calling monclient init 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.427+0000 7fd7b16f1640 10 monclient: init 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac0ab9e0 con 0x7fd7ac0a98f0 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac1d7830 con 0x7fd7ac13f710 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac1d7500 con 0x7fd7ac1430c0 2024-03-25T17:41:40.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:40.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.428+0000 7fd7b16f1640 10 monclient(hunting): authenticate will time out at 6103.656232s 2024-03-25T17:41:40.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.439+0000 7fd7ab7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fd7ac1430c0 0x7fd7ac134080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:39294/0 (socket says 172.21.15.44:39294) 2024-03-25T17:41:40.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.439+0000 7fd7ab7fe640 1 -- 172.21.15.44:0/1028498256 learned_addr learned my addr 172.21.15.44:0/1028498256 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:40.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.444+0000 7fd7a37fe640 10 client.?.objecter ms_handle_connect 0x7fd7ac1430c0 2024-03-25T17:41:40.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.444+0000 7fd7a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:40.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.444+0000 7fd7a37fe640 10 client.?.objecter ms_handle_connect 0x7fd7ac13f710 2024-03-25T17:41:40.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.444+0000 7fd7a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:40.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.450+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576784837 0 0) 0x7fd7ac1d7830 con 0x7fd7ac13f710 2024-03-25T17:41:40.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.450+0000 7fd7a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:40.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.450+0000 7fd7a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:40.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.450+0000 7fd7a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:40.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.450+0000 7fd7a37fe640 10 monclient(hunting): my global_id is 4339 2024-03-25T17:41:40.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.450+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c004eb0 con 0x7fd7ac13f710 2024-03-25T17:41:40.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.450+0000 7fd7a37fe640 10 client.?.objecter ms_handle_connect 0x7fd7ac0a98f0 2024-03-25T17:41:40.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.450+0000 7fd7a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:40.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.454+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454841943 0 0) 0x7fd7ac1d7500 con 0x7fd7ac1430c0 2024-03-25T17:41:40.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.454+0000 7fd7a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:40.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.454+0000 7fd7a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:40.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.454+0000 7fd7a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:40.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.454+0000 7fd7a37fe640 10 monclient(hunting): my global_id is 4334 2024-03-25T17:41:40.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.454+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7ac1d7830 con 0x7fd7ac1430c0 2024-03-25T17:41:40.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.459+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191138699 0 0) 0x7fd78c004eb0 con 0x7fd7ac13f710 2024-03-25T17:41:40.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.459+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7ac1d7500 con 0x7fd7ac13f710 2024-03-25T17:41:40.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.459+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd794001ae0 con 0x7fd7ac13f710 2024-03-25T17:41:40.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.459+0000 7fd7a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:40.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.459+0000 7fd7a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:40.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.459+0000 7fd7a37fe640 10 monclient(hunting): dump: 2024-03-25T17:41:40.459 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:40.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.459+0000 7fd7a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:41:40.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.461+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185421675 0 0) 0x7fd7ac0ab9e0 con 0x7fd7ac0a98f0 2024-03-25T17:41:40.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.461+0000 7fd7a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:40.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.461+0000 7fd7a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:40.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.461+0000 7fd7a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:40.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.461+0000 7fd7a37fe640 10 monclient(hunting): my global_id is 4350 2024-03-25T17:41:40.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.461+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c004eb0 con 0x7fd7ac0a98f0 2024-03-25T17:41:40.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 258352376 0 0) 0x7fd7ac1d7500 con 0x7fd7ac13f710 2024-03-25T17:41:40.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 >> v1:172.21.15.44:6790/0 conn(0x7fd7ac1430c0 legacy=0x7fd7ac134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:40.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 >> v1:172.21.15.44:6789/0 conn(0x7fd7ac0a98f0 legacy=0x7fd7ac137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:40.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:41:40.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 1 monclient: found mon.b 2024-03-25T17:41:40.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:40.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:40.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac134790 con 0x7fd7ac13f710 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 10 monclient: _finish_auth 0 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 5 monclient: authenticate success, global_id 4339 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 10 monclient: _renew_subs 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac1349e0 con 0x7fd7ac13f710 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 10 monclient: _renew_subs 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac138b10 con 0x7fd7ac13f710 2024-03-25T17:41:40.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.467+0000 7fd7b16f1640 1 librados: init done 2024-03-25T17:41:40.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.468+0000 7fd7b16f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:40.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.468+0000 7fd7b16f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:40.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.468+0000 7fd7b16f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:40.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.468+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ac1d7690 con 0x7fd7ac13f710 2024-03-25T17:41:40.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.737+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd794004a90 con 0x7fd7ac13f710 2024-03-25T17:41:40.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.737+0000 7fd7a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:40.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.737+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd794004c00 con 0x7fd7ac13f710 2024-03-25T17:41:40.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.737+0000 7fd7a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:40.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.737+0000 7fd7a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:41:40.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.737+0000 7fd7a37fe640 10 monclient: dump: 2024-03-25T17:41:40.738 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:40.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.737+0000 7fd7a9ffb640 4 set_mon_vals no callback set 2024-03-25T17:41:40.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.738+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fd79401abd0 con 0x7fd7ac13f710 2024-03-25T17:41:40.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.739+0000 7fd7aaffd640 10 monclient: get_auth_request con 0x7fd78c06de00 auth_method 0 2024-03-25T17:41:40.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.739+0000 7fd7a37fe640 10 client.4339.objecter ms_handle_connect 0x7fd78c06de00 2024-03-25T17:41:40.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.743+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fd794054730 con 0x7fd7ac13f710 2024-03-25T17:41:40.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.743+0000 7fd7a37fe640 10 client.4339.objecter ms_dispatch 0x7fd7ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:41:40.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.743+0000 7fd7a37fe640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:41:40.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.743+0000 7fd7a37fe640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:41:40.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.743+0000 7fd7a37fe640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T17:41:40.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.743+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd794050240 con 0x7fd7ac13f710 2024-03-25T17:41:40.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.743+0000 7fd7a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:40.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:40.743+0000 7fd7a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:41:41.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.002+0000 7fd7b16f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:41.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.002+0000 7fd7b16f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:41.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.002+0000 7fd7b16f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:41:41.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.002+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7ac14f320 con 0x7fd7ac13f710 2024-03-25T17:41:41.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.032+0000 7fd7a37fe640 1 -- 172.21.15.44:0/1028498256 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fd794052170 con 0x7fd7ac13f710 2024-03-25T17:41:41.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.032+0000 7fd7a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:41.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.032+0000 7fd7a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:41:41.033 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:41:41.033 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 10 librados: watch_flush enter 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 10 librados: watch_flush exit 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 10 client.4339.objecter successfully canceled tick 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 >> v1:172.21.15.44:6800/3257043132 conn(0x7fd78c06de00 legacy=0x7fd78c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 10 monclient: shutdown 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 >> v1:172.21.15.111:6789/0 conn(0x7fd7ac13f710 legacy=0x7fd7ac1d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 shutdown_connections 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 >> 172.21.15.44:0/1028498256 conn(0x7fd7ac0b7ea0 msgr2=0x7fd7ac13e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:41.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 shutdown_connections 2024-03-25T17:41:41.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 1 -- 172.21.15.44:0/1028498256 wait complete. 2024-03-25T17:41:41.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:41.034+0000 7fd7b16f1640 1 librados: shutdown 2024-03-25T17:41:41.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:41:47.852 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:41:47.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.965+0000 7ff71b748640 10 monclient: get_monmap_and_config 2024-03-25T17:41:47.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.965+0000 7ff71b748640 10 monclient: build_initial_monmap 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.965+0000 7ff71b748640 10 monclient: monmap: 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:41:47.966318+0000 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:41:47.966318+0000 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:41:47.966 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.966+0000 7ff71b748640 1 Processor -- start 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.966+0000 7ff71b748640 1 -- start start 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.966+0000 7ff71b748640 10 monclient: init 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.966+0000 7ff71b748640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.966+0000 7ff71b748640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.966+0000 7ff71b748640 10 monclient(hunting): picked mon.noname-c con 0x7ff7140a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.967+0000 7ff71b748640 10 monclient(hunting): picked mon.noname-a con 0x7ff71413f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.967+0000 7ff71b748640 10 monclient(hunting): picked mon.noname-b con 0x7ff7141430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.967+0000 7ff71b748640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714142520 con 0x7ff71413f710 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.967+0000 7ff71b748640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7140b28a0 con 0x7ff7140a98f0 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.967+0000 7ff71b748640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7140b2a30 con 0x7ff7141430c0 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.967+0000 7ff71b748640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:47.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.967+0000 7ff71b748640 10 monclient(hunting): authenticate will time out at 6111.194987s 2024-03-25T17:41:47.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.972+0000 7ff719ce5640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff7141430c0 0x7ff71414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:44006/0 (socket says 172.21.15.44:44006) 2024-03-25T17:41:47.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.972+0000 7ff719ce5640 1 -- 172.21.15.44:0/3649668598 learned_addr learned my addr 172.21.15.44:0/3649668598 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:47.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.980+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708599619 0 0) 0x7ff7140b28a0 con 0x7ff7140a98f0 2024-03-25T17:41:47.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.980+0000 7ff70bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:41:47.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.980+0000 7ff70bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:47.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.980+0000 7ff70bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:47.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.980+0000 7ff70bfff640 10 monclient(hunting): my global_id is 4342 2024-03-25T17:41:47.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.980+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff700003520 con 0x7ff7140a98f0 2024-03-25T17:41:47.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3068183227 0 0) 0x7ff700003520 con 0x7ff7140a98f0 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 >> v1:172.21.15.44:6790/0 conn(0x7ff7141430c0 legacy=0x7ff71414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 >> v1:172.21.15.44:6789/0 conn(0x7ff71413f710 legacy=0x7ff7141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 1 monclient: found mon.noname-c 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7140b44c0 con 0x7ff7140a98f0 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 10 monclient: _finish_auth 0 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff710001690 con 0x7ff7140a98f0 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:47.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff70bfff640 10 monclient: dump: 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff71b748640 5 monclient: authenticate success, global_id 4342 2024-03-25T17:41:47.986 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.984+0000 7ff71b748640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:47.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.994+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff710001860 con 0x7ff7140a98f0 2024-03-25T17:41:47.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.994+0000 7ff70bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:47.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.994+0000 7ff70bfff640 1 -- 172.21.15.44:0/3649668598 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff710001cb0 con 0x7ff7140a98f0 2024-03-25T17:41:47.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.994+0000 7ff70bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.994+0000 7ff70bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.994+0000 7ff70bfff640 10 monclient: dump: 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.994+0000 7ff71b748640 10 monclient: get_monmap_and_config success 2024-03-25T17:41:47.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.994+0000 7ff71b748640 4 set_mon_vals no callback set 2024-03-25T17:41:47.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.995+0000 7ff71b748640 10 monclient: shutdown 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.995+0000 7ff71b748640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.995+0000 7ff71b748640 1 -- 172.21.15.44:0/3649668598 >> v1:172.21.15.111:6789/0 conn(0x7ff7140a98f0 legacy=0x7ff7140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.995+0000 7ff71b748640 1 -- 172.21.15.44:0/3649668598 shutdown_connections 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.995+0000 7ff71b748640 1 -- 172.21.15.44:0/3649668598 >> 172.21.15.44:0/3649668598 conn(0x7ff7140b7ea0 msgr2=0x7ff7140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.995+0000 7ff71b748640 1 -- 172.21.15.44:0/3649668598 shutdown_connections 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.995+0000 7ff71b748640 1 -- 172.21.15.44:0/3649668598 wait complete. 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 10 monclient: build_initial_monmap 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 10 monclient: monmap: 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:41:47.998 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 1 librados: starting msgr at 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 1 librados: starting objecter 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 1 Processor -- start 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 1 -- start start 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 1 librados: setting wanted keys 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 1 librados: calling monclient init 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.996+0000 7ff71b748640 10 monclient: init 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:47.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:41:48.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 10 monclient(hunting): picked mon.noname-c con 0x7ff7140a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:48.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 10 monclient(hunting): picked mon.noname-a con 0x7ff71413f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:48.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 10 monclient(hunting): picked mon.noname-b con 0x7ff7141430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:48.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714137320 con 0x7ff71413f710 2024-03-25T17:41:48.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7141368a0 con 0x7ff7141430c0 2024-03-25T17:41:48.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714136a30 con 0x7ff7140a98f0 2024-03-25T17:41:48.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:48.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:47.997+0000 7ff71b748640 10 monclient(hunting): authenticate will time out at 6111.225125s 2024-03-25T17:41:48.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.004+0000 7ff7194e4640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff7140a98f0 0x7ff71413b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:44038/0 (socket says 172.21.15.44:44038) 2024-03-25T17:41:48.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.004+0000 7ff7194e4640 1 -- 172.21.15.44:0/1353063387 learned_addr learned my addr 172.21.15.44:0/1353063387 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:48.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.006+0000 7ff709ffb640 10 client.?.objecter ms_handle_connect 0x7ff7140a98f0 2024-03-25T17:41:48.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.006+0000 7ff709ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:48.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.016+0000 7ff709ffb640 10 client.?.objecter ms_handle_connect 0x7ff7141430c0 2024-03-25T17:41:48.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.016+0000 7ff709ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:48.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.025+0000 7ff709ffb640 10 client.?.objecter ms_handle_connect 0x7ff71413f710 2024-03-25T17:41:48.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.025+0000 7ff709ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:48.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.029+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978146956 0 0) 0x7ff714136a30 con 0x7ff7140a98f0 2024-03-25T17:41:48.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.029+0000 7ff709ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:41:48.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.029+0000 7ff709ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:48.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.029+0000 7ff709ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:48.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.029+0000 7ff709ffb640 10 monclient(hunting): my global_id is 4340 2024-03-25T17:41:48.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.029+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f8004eb0 con 0x7ff7140a98f0 2024-03-25T17:41:48.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.041+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175510735 0 0) 0x7ff7141368a0 con 0x7ff7141430c0 2024-03-25T17:41:48.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.041+0000 7ff709ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:41:48.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.041+0000 7ff709ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:48.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.041+0000 7ff709ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:48.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.041+0000 7ff709ffb640 10 monclient(hunting): my global_id is 4345 2024-03-25T17:41:48.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.041+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff714136a30 con 0x7ff7141430c0 2024-03-25T17:41:48.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.043+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150411286 0 0) 0x7ff6f8004eb0 con 0x7ff7140a98f0 2024-03-25T17:41:48.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.043+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7141368a0 con 0x7ff7140a98f0 2024-03-25T17:41:48.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.043+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff710001c30 con 0x7ff7140a98f0 2024-03-25T17:41:48.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.043+0000 7ff709ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:48.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.043+0000 7ff709ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:41:48.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.043+0000 7ff709ffb640 10 monclient(hunting): dump: 2024-03-25T17:41:48.044 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:48.044 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:48.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.043+0000 7ff709ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:41:48.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.046+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493577818 0 0) 0x7ff714137320 con 0x7ff71413f710 2024-03-25T17:41:48.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.046+0000 7ff709ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:41:48.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.046+0000 7ff709ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:48.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.046+0000 7ff709ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:48.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.046+0000 7ff709ffb640 10 monclient(hunting): my global_id is 4353 2024-03-25T17:41:48.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.046+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f8004eb0 con 0x7ff71413f710 2024-03-25T17:41:48.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2278654382 0 0) 0x7ff7141368a0 con 0x7ff7140a98f0 2024-03-25T17:41:48.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 >> v1:172.21.15.111:6789/0 conn(0x7ff7141430c0 legacy=0x7ff7141e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:48.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 >> v1:172.21.15.44:6789/0 conn(0x7ff71413f710 legacy=0x7ff714134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:48.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:41:48.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 1 monclient: found mon.c 2024-03-25T17:41:48.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:48.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:41:48.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff714139690 con 0x7ff7140a98f0 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 10 monclient: _finish_auth 0 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff709ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 5 monclient: authenticate success, global_id 4340 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 10 monclient: _renew_subs 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff714139940 con 0x7ff7140a98f0 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 10 monclient: _renew_subs 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff714137990 con 0x7ff7140a98f0 2024-03-25T17:41:48.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.057+0000 7ff71b748640 1 librados: init done 2024-03-25T17:41:48.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.058+0000 7ff71b748640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:48.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.058+0000 7ff71b748640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:48.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.058+0000 7ff71b748640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:41:48.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.058+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7140b3ef0 con 0x7ff7140a98f0 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.086+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff710003940 con 0x7ff7140a98f0 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.086+0000 7ff709ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.086+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff710003d10 con 0x7ff7140a98f0 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.086+0000 7ff709ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.086+0000 7ff709ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.086+0000 7ff709ffb640 10 monclient: dump: 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:48.087 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:48.088 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:48.088 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:48.088 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:48.088 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:48.088 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:48.088 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:48.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.086+0000 7ff70b7fe640 4 set_mon_vals no callback set 2024-03-25T17:41:48.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.087+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7ff710003f30 con 0x7ff7140a98f0 2024-03-25T17:41:48.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.087+0000 7ff718ce3640 10 monclient: get_auth_request con 0x7ff6f806de10 auth_method 0 2024-03-25T17:41:48.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.088+0000 7ff709ffb640 10 client.4340.objecter ms_handle_connect 0x7ff6f806de10 2024-03-25T17:41:48.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.090+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7ff7100548f0 con 0x7ff7140a98f0 2024-03-25T17:41:48.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.090+0000 7ff709ffb640 10 client.4340.objecter ms_dispatch 0x7ff714000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:41:48.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.090+0000 7ff709ffb640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:41:48.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.090+0000 7ff709ffb640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:41:48.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.090+0000 7ff709ffb640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T17:41:48.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.388+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff710050400 con 0x7ff7140a98f0 2024-03-25T17:41:48.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.388+0000 7ff709ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:48.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.388+0000 7ff709ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:41:48.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.612+0000 7ff71b748640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:48.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.612+0000 7ff71b748640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:48.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.612+0000 7ff71b748640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:41:48.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.612+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff714139140 con 0x7ff7140a98f0 2024-03-25T17:41:48.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.725+0000 7ff709ffb640 1 -- 172.21.15.44:0/1353063387 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7ff714139140 con 0x7ff7140a98f0 2024-03-25T17:41:48.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.725+0000 7ff709ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:48.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.725+0000 7ff709ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:41:48.725 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:41:48.726 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 10 librados: watch_flush enter 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 10 librados: watch_flush exit 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 10 client.4340.objecter successfully canceled tick 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 >> v1:172.21.15.44:6800/3257043132 conn(0x7ff6f806de10 legacy=0x7ff6f80702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 10 monclient: shutdown 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 >> v1:172.21.15.44:6790/0 conn(0x7ff7140a98f0 legacy=0x7ff71413b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 shutdown_connections 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 >> 172.21.15.44:0/1353063387 conn(0x7ff7140b7ea0 msgr2=0x7ff71413c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:48.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 shutdown_connections 2024-03-25T17:41:48.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.727+0000 7ff71b748640 1 -- 172.21.15.44:0/1353063387 wait complete. 2024-03-25T17:41:48.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:48.728+0000 7ff71b748640 1 librados: shutdown 2024-03-25T17:41:48.742 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:41:55.549 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:41:55.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.664+0000 7ff4010e6640 10 monclient: get_monmap_and_config 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.664+0000 7ff4010e6640 10 monclient: build_initial_monmap 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.664+0000 7ff4010e6640 10 monclient: monmap: 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:41:55.665350+0000 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:41:55.665350+0000 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:41:55.665 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:41:55.666 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:55.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 1 Processor -- start 2024-03-25T17:41:55.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 1 -- start start 2024-03-25T17:41:55.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 10 monclient: init 2024-03-25T17:41:55.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:55.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:41:55.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff3fc0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:55.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 10 monclient(hunting): picked mon.noname-c con 0x7ff3fc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:55.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff3fc1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:55.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc142520 con 0x7ff3fc1430c0 2024-03-25T17:41:55.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b28a0 con 0x7ff3fc13f710 2024-03-25T17:41:55.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b2a30 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:55.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.665+0000 7ff4010e6640 10 monclient(hunting): authenticate will time out at 6118.893631s 2024-03-25T17:41:55.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.674+0000 7ff3fad9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff3fc0a98f0 0x7ff3fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:53536/0 (socket says 172.21.15.44:53536) 2024-03-25T17:41:55.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.674+0000 7ff3fad9d640 1 -- 172.21.15.44:0/3986774261 learned_addr learned my addr 172.21.15.44:0/3986774261 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:55.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.685+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563874057 0 0) 0x7ff3fc142520 con 0x7ff3fc1430c0 2024-03-25T17:41:55.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.685+0000 7ff3fa59c640 10 monclient(hunting): _negotiate 2024-03-25T17:41:55.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.685+0000 7ff3fa59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:55.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.685+0000 7ff3fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:55.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.685+0000 7ff3fa59c640 10 monclient(hunting): my global_id is 4356 2024-03-25T17:41:55.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.685+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e8003520 con 0x7ff3fc1430c0 2024-03-25T17:41:55.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.692+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117540729 0 0) 0x7ff3fc0b2a30 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.692+0000 7ff3fa59c640 10 monclient(hunting): _negotiate 2024-03-25T17:41:55.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.692+0000 7ff3fa59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:55.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.692+0000 7ff3fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:55.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.692+0000 7ff3fa59c640 10 monclient(hunting): my global_id is 4343 2024-03-25T17:41:55.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.692+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc142520 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.697+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129529705 0 0) 0x7ff3fc0b28a0 con 0x7ff3fc13f710 2024-03-25T17:41:55.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.697+0000 7ff3fa59c640 10 monclient(hunting): _negotiate 2024-03-25T17:41:55.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.697+0000 7ff3fa59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:55.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.697+0000 7ff3fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:55.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.697+0000 7ff3fa59c640 10 monclient(hunting): my global_id is 4348 2024-03-25T17:41:55.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.697+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc0b2a30 con 0x7ff3fc13f710 2024-03-25T17:41:55.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1916612633 0 0) 0x7ff3fc142520 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 >> v1:172.21.15.111:6789/0 conn(0x7ff3fc13f710 legacy=0x7ff3fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 >> v1:172.21.15.44:6789/0 conn(0x7ff3fc1430c0 legacy=0x7ff3fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 10 monclient: _finish_hunting 0 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 1 monclient: found mon.noname-b 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc0b44c0 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 10 monclient: _finish_auth 0 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 10 monclient: _check_auth_tickets 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff3ec001670 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:41:55.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff3fa59c640 10 monclient: dump: 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff4010e6640 5 monclient: authenticate success, global_id 4343 2024-03-25T17:41:55.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.702+0000 7ff4010e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:41:55.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.723+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ec0017d0 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.723+0000 7ff3fa59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:55.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.723+0000 7ff3fa59c640 1 -- 172.21.15.44:0/3986774261 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff3ec001be0 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.723+0000 7ff3fa59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.723+0000 7ff3fa59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.723+0000 7ff3fa59c640 10 monclient: dump: 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:55.724 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.723+0000 7ff4010e6640 10 monclient: get_monmap_and_config success 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.723+0000 7ff4010e6640 4 set_mon_vals no callback set 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.724+0000 7ff4010e6640 10 monclient: shutdown 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.724+0000 7ff4010e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.724+0000 7ff4010e6640 1 -- 172.21.15.44:0/3986774261 >> v1:172.21.15.44:6790/0 conn(0x7ff3fc0a98f0 legacy=0x7ff3fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.724+0000 7ff4010e6640 1 -- 172.21.15.44:0/3986774261 shutdown_connections 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.724+0000 7ff4010e6640 1 -- 172.21.15.44:0/3986774261 >> 172.21.15.44:0/3986774261 conn(0x7ff3fc0b7ea0 msgr2=0x7ff3fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.724+0000 7ff4010e6640 1 -- 172.21.15.44:0/3986774261 shutdown_connections 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.724+0000 7ff4010e6640 1 -- 172.21.15.44:0/3986774261 wait complete. 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.725+0000 7ff4010e6640 10 monclient: build_initial_monmap 2024-03-25T17:41:55.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.725+0000 7ff4010e6640 10 monclient: monmap: 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.725+0000 7ff4010e6640 1 librados: starting msgr at 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.725+0000 7ff4010e6640 1 librados: starting objecter 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.725+0000 7ff4010e6640 1 Processor -- start 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 1 -- start start 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 1 librados: setting wanted keys 2024-03-25T17:41:55.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 1 librados: calling monclient init 2024-03-25T17:41:55.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 10 monclient: init 2024-03-25T17:41:55.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:41:55.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:41:55.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 10 monclient(hunting): picked mon.noname-c con 0x7ff3fc0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:41:55.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff3fc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:41:55.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff3fc1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:41:55.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc136e60 con 0x7ff3fc1430c0 2024-03-25T17:41:55.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc1381a0 con 0x7ff3fc13f710 2024-03-25T17:41:55.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc1370b0 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:41:55.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.726+0000 7ff4010e6640 10 monclient(hunting): authenticate will time out at 6118.954319s 2024-03-25T17:41:55.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.741+0000 7ff3fb59e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff3fc1430c0 0x7ff3fc13ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:38430/0 (socket says 172.21.15.44:38430) 2024-03-25T17:41:55.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.741+0000 7ff3fb59e640 1 -- 172.21.15.44:0/1433854778 learned_addr learned my addr 172.21.15.44:0/1433854778 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:41:55.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.742+0000 7ff3f37fe640 10 client.?.objecter ms_handle_connect 0x7ff3fc1430c0 2024-03-25T17:41:55.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.742+0000 7ff3f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:55.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.743+0000 7ff3f37fe640 10 client.?.objecter ms_handle_connect 0x7ff3fc0a98f0 2024-03-25T17:41:55.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.743+0000 7ff3f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:55.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.750+0000 7ff3f37fe640 10 client.?.objecter ms_handle_connect 0x7ff3fc13f710 2024-03-25T17:41:55.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.750+0000 7ff3f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.771+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154175822 0 0) 0x7ff3fc136e60 con 0x7ff3fc1430c0 2024-03-25T17:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.771+0000 7ff3f37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.771+0000 7ff3f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.771+0000 7ff3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.771+0000 7ff3f37fe640 10 monclient(hunting): my global_id is 4359 2024-03-25T17:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.771+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc004eb0 con 0x7ff3fc1430c0 2024-03-25T17:41:55.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953501209 0 0) 0x7ff3fc1370b0 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:55.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:55.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:55.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T17:41:55.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc136e60 con 0x7ff3fc0a98f0 2024-03-25T17:41:55.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994687297 0 0) 0x7ff3fc1381a0 con 0x7ff3fc13f710 2024-03-25T17:41:55.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:41:55.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:41:55.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:41:55.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 10 monclient(hunting): my global_id is 4351 2024-03-25T17:41:55.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.776+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc1370b0 con 0x7ff3fc13f710 2024-03-25T17:41:55.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.793+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 75512187 0 0) 0x7ff3dc004eb0 con 0x7ff3fc1430c0 2024-03-25T17:41:55.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.793+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3fc1381a0 con 0x7ff3fc1430c0 2024-03-25T17:41:55.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.793+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff3f4001c30 con 0x7ff3fc1430c0 2024-03-25T17:41:55.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.793+0000 7ff3f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:55.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.793+0000 7ff3f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.793+0000 7ff3f37fe640 10 monclient(hunting): dump: 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:55.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.793+0000 7ff3f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.805+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3367253020 0 0) 0x7ff3fc1370b0 con 0x7ff3fc13f710 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.805+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3dc004eb0 con 0x7ff3fc13f710 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.805+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff3e4001830 con 0x7ff3fc13f710 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.805+0000 7ff3f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.805+0000 7ff3f37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.805+0000 7ff3f37fe640 10 monclient(hunting): dump: 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:55.806 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:55.807 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:55.807 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:55.807 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:55.807 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:55.807 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:55.807 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:55.807 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703995606 0 0) 0x7ff3fc1381a0 con 0x7ff3fc1430c0 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 >> v1:172.21.15.44:6790/0 conn(0x7ff3fc0a98f0 legacy=0x7ff3fc137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 >> v1:172.21.15.111:6789/0 conn(0x7ff3fc13f710 legacy=0x7ff3fc13a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 1 monclient: found mon.a 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc137390 con 0x7ff3fc1430c0 2024-03-25T17:41:55.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 10 monclient: _finish_auth 0 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff3f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 5 monclient: authenticate success, global_id 4359 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 10 monclient: _renew_subs 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc1376d0 con 0x7ff3fc1430c0 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 10 monclient: _renew_subs 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:55.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc135530 con 0x7ff3fc1430c0 2024-03-25T17:41:55.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.821+0000 7ff4010e6640 1 librados: init done 2024-03-25T17:41:55.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.822+0000 7ff4010e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:55.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.822+0000 7ff4010e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:55.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.822+0000 7ff4010e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:55.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:55.822+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3fc0b3ef0 con 0x7ff3fc1430c0 2024-03-25T17:41:56.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.180+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f40020e0 con 0x7ff3fc1430c0 2024-03-25T17:41:56.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.180+0000 7ff3f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.180+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7ff3f4003b70 con 0x7ff3fc1430c0 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.180+0000 7ff3f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.180+0000 7ff3f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.180+0000 7ff3f37fe640 10 monclient: dump: 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:41:56.181 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:41:56.182 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:41:56.182 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:41:56.182 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:41:56.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.180+0000 7ff3f9d9b640 4 set_mon_vals no callback set 2024-03-25T17:41:56.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.181+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7ff3f4003d90 con 0x7ff3fc1430c0 2024-03-25T17:41:56.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.181+0000 7ff3fad9d640 10 monclient: get_auth_request con 0x7ff3dc06df20 auth_method 0 2024-03-25T17:41:56.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.181+0000 7ff3f37fe640 10 client.4359.objecter ms_handle_connect 0x7ff3dc06df20 2024-03-25T17:41:56.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.187+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7ff3f4054a50 con 0x7ff3fc1430c0 2024-03-25T17:41:56.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.187+0000 7ff3f37fe640 10 client.4359.objecter ms_dispatch 0x7ff3fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:41:56.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.187+0000 7ff3f37fe640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:41:56.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.187+0000 7ff3f37fe640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:41:56.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.187+0000 7ff3f37fe640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-25T17:41:56.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.187+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3f4050390 con 0x7ff3fc1430c0 2024-03-25T17:41:56.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.187+0000 7ff3f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:41:56.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.187+0000 7ff3f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:41:56.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.421+0000 7ff4010e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:56.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.421+0000 7ff4010e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:56.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.421+0000 7ff4010e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:41:56.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.421+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3fc137520 con 0x7ff3fc1430c0 2024-03-25T17:41:56.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.489+0000 7ff3f37fe640 1 -- 172.21.15.44:0/1433854778 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7ff3f4052490 con 0x7ff3fc1430c0 2024-03-25T17:41:56.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.489+0000 7ff3f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:41:56.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.489+0000 7ff3f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:41:56.490 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:41:56.490 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 10 librados: watch_flush enter 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 10 librados: watch_flush exit 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 10 client.4359.objecter successfully canceled tick 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 >> v1:172.21.15.44:6800/3257043132 conn(0x7ff3dc06df20 legacy=0x7ff3dc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 10 monclient: shutdown 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 >> v1:172.21.15.44:6789/0 conn(0x7ff3fc1430c0 legacy=0x7ff3fc13ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 shutdown_connections 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 >> 172.21.15.44:0/1433854778 conn(0x7ff3fc0b7ea0 msgr2=0x7ff3fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:41:56.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 shutdown_connections 2024-03-25T17:41:56.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 1 -- 172.21.15.44:0/1433854778 wait complete. 2024-03-25T17:41:56.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:41:56.492+0000 7ff4010e6640 1 librados: shutdown 2024-03-25T17:41:56.506 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:41:57.634 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:14.990 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:57.640 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:14.996 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:57.673 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:15.029 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:57.685 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:15.005 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:57.695 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:15.015 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:57.704 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:15.060 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:57.713 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:15.069 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:57.724 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:15.044 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:57.757 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:15.113 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:03.312 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.429+0000 7f504165c640 10 monclient: get_monmap_and_config 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.429+0000 7f504165c640 10 monclient: build_initial_monmap 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.429+0000 7f504165c640 10 monclient: monmap: 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:42:03.430255+0000 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:42:03.430255+0000 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:03.430 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:42:03.431 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:42:03.431 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.430+0000 7f504165c640 1 Processor -- start 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.430+0000 7f504165c640 1 -- start start 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.430+0000 7f504165c640 10 monclient: init 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 10 monclient(hunting): picked mon.noname-c con 0x7f503c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 10 monclient(hunting): picked mon.noname-b con 0x7f503c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 10 monclient(hunting): picked mon.noname-a con 0x7f503c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c142520 con 0x7f503c1430c0 2024-03-25T17:42:03.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0b28a0 con 0x7f503c0a98f0 2024-03-25T17:42:03.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0b2a30 con 0x7f503c13f710 2024-03-25T17:42:03.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:03.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.431+0000 7f504165c640 10 monclient(hunting): authenticate will time out at 6126.659059s 2024-03-25T17:42:03.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.437+0000 7f503a7fc640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f503c13f710 0x7f503c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:57510/0 (socket says 172.21.15.44:57510) 2024-03-25T17:42:03.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.437+0000 7f503a7fc640 1 -- 172.21.15.44:0/1197195326 learned_addr learned my addr 172.21.15.44:0/1197195326 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:03.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.446+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396788519 0 0) 0x7f503c0b2a30 con 0x7f503c13f710 2024-03-25T17:42:03.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.446+0000 7f5039ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:42:03.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.446+0000 7f5039ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:03.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.446+0000 7f5039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:03.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.446+0000 7f5039ffb640 10 monclient(hunting): my global_id is 4349 2024-03-25T17:42:03.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.446+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5024003520 con 0x7f503c13f710 2024-03-25T17:42:03.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619678831 0 0) 0x7f503c142520 con 0x7f503c1430c0 2024-03-25T17:42:03.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:42:03.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:03.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:03.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient(hunting): my global_id is 4362 2024-03-25T17:42:03.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c0b2a30 con 0x7f503c1430c0 2024-03-25T17:42:03.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3372264668 0 0) 0x7f5024003520 con 0x7f503c13f710 2024-03-25T17:42:03.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 >> v1:172.21.15.111:6789/0 conn(0x7f503c0a98f0 legacy=0x7f503c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 >> v1:172.21.15.44:6789/0 conn(0x7f503c1430c0 legacy=0x7f503c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 1 monclient: found mon.noname-b 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c0b44c0 con 0x7f503c13f710 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient: _finish_auth 0 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5030001690 con 0x7f503c13f710 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.456+0000 7f5039ffb640 10 monclient: dump: 2024-03-25T17:42:03.458 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.457+0000 7f504165c640 5 monclient: authenticate success, global_id 4349 2024-03-25T17:42:03.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.457+0000 7f504165c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.473+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50300017f0 con 0x7f503c13f710 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.473+0000 7f5039ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.473+0000 7f5039ffb640 1 -- 172.21.15.44:0/1197195326 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5030001c00 con 0x7f503c13f710 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.473+0000 7f5039ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.473+0000 7f5039ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.473+0000 7f5039ffb640 10 monclient: dump: 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:03.474 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:03.475 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:03.475 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:03.475 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:03.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.473+0000 7f504165c640 10 monclient: get_monmap_and_config success 2024-03-25T17:42:03.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.473+0000 7f504165c640 4 set_mon_vals no callback set 2024-03-25T17:42:03.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.474+0000 7f504165c640 10 monclient: shutdown 2024-03-25T17:42:03.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.474+0000 7f504165c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:03.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.474+0000 7f504165c640 1 -- 172.21.15.44:0/1197195326 >> v1:172.21.15.44:6790/0 conn(0x7f503c13f710 legacy=0x7f503c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:03.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.474+0000 7f504165c640 1 -- 172.21.15.44:0/1197195326 shutdown_connections 2024-03-25T17:42:03.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.474+0000 7f504165c640 1 -- 172.21.15.44:0/1197195326 >> 172.21.15.44:0/1197195326 conn(0x7f503c0b7ea0 msgr2=0x7f503c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:03.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.474+0000 7f504165c640 1 -- 172.21.15.44:0/1197195326 shutdown_connections 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.474+0000 7f504165c640 1 -- 172.21.15.44:0/1197195326 wait complete. 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.475+0000 7f504165c640 10 monclient: build_initial_monmap 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.475+0000 7f504165c640 10 monclient: monmap: 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.475+0000 7f504165c640 1 librados: starting msgr at 2024-03-25T17:42:03.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.475+0000 7f504165c640 1 librados: starting objecter 2024-03-25T17:42:03.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 1 Processor -- start 2024-03-25T17:42:03.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 1 -- start start 2024-03-25T17:42:03.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 1 librados: setting wanted keys 2024-03-25T17:42:03.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 1 librados: calling monclient init 2024-03-25T17:42:03.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 10 monclient: init 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 10 monclient(hunting): picked mon.noname-a con 0x7f503c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 10 monclient(hunting): picked mon.noname-c con 0x7f503c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 10 monclient(hunting): picked mon.noname-b con 0x7f503c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c1390d0 con 0x7f503c0a98f0 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c136cc0 con 0x7f503c1430c0 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c137a70 con 0x7f503c13f710 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:03.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.476+0000 7f504165c640 10 monclient(hunting): authenticate will time out at 6126.704649s 2024-03-25T17:42:03.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.481+0000 7f503b7fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f503c1430c0 0x7f503c1e2f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:57614/0 (socket says 172.21.15.44:57614) 2024-03-25T17:42:03.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.481+0000 7f503b7fe640 1 -- 172.21.15.44:0/3267197077 learned_addr learned my addr 172.21.15.44:0/3267197077 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:03.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.485+0000 7f50237fe640 10 client.?.objecter ms_handle_connect 0x7f503c1430c0 2024-03-25T17:42:03.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.485+0000 7f50237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:03.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.490+0000 7f50237fe640 10 client.?.objecter ms_handle_connect 0x7f503c13f710 2024-03-25T17:42:03.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.490+0000 7f50237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.494+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335077829 0 0) 0x7f503c137a70 con 0x7f503c13f710 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.494+0000 7f50237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.494+0000 7f50237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.494+0000 7f50237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.494+0000 7f50237fe640 10 monclient(hunting): my global_id is 4352 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.494+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5028004eb0 con 0x7f503c13f710 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.495+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129531137 0 0) 0x7f503c136cc0 con 0x7f503c1430c0 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.495+0000 7f50237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:42:03.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.495+0000 7f50237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:03.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.495+0000 7f50237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:03.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.495+0000 7f50237fe640 10 monclient(hunting): my global_id is 4354 2024-03-25T17:42:03.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.495+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c137a70 con 0x7f503c1430c0 2024-03-25T17:42:03.497 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.496+0000 7f50237fe640 10 client.?.objecter ms_handle_connect 0x7f503c0a98f0 2024-03-25T17:42:03.497 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.496+0000 7f50237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:03.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639342517 0 0) 0x7f503c1390d0 con 0x7f503c0a98f0 2024-03-25T17:42:03.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 10 monclient(hunting): _negotiate 2024-03-25T17:42:03.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:03.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:03.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 10 monclient(hunting): my global_id is 4365 2024-03-25T17:42:03.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c136cc0 con 0x7f503c0a98f0 2024-03-25T17:42:03.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3317920598 0 0) 0x7f5028004eb0 con 0x7f503c13f710 2024-03-25T17:42:03.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f503c1390d0 con 0x7f503c13f710 2024-03-25T17:42:03.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5030001b00 con 0x7f503c13f710 2024-03-25T17:42:03.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:03.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:42:03.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 10 monclient(hunting): dump: 2024-03-25T17:42:03.507 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:03.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.506+0000 7f50237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.511+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 987308180 0 0) 0x7f503c137a70 con 0x7f503c1430c0 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.511+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5028004eb0 con 0x7f503c1430c0 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.511+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5034001d20 con 0x7f503c1430c0 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.511+0000 7f50237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.511+0000 7f50237fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.511+0000 7f50237fe640 10 monclient(hunting): dump: 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.511 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.512 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:03.512 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:03.512 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:03.512 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:03.512 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:03.512 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:03.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1000011547 0 0) 0x7f503c1390d0 con 0x7f503c13f710 2024-03-25T17:42:03.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 >> v1:172.21.15.111:6789/0 conn(0x7f503c1430c0 legacy=0x7f503c1e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:03.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 >> v1:172.21.15.44:6789/0 conn(0x7f503c0a98f0 legacy=0x7f503c1dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:03.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 10 monclient: _finish_hunting 0 2024-03-25T17:42:03.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 1 monclient: found mon.c 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c137f90 con 0x7f503c13f710 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 10 monclient: _finish_auth 0 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 10 monclient: _check_auth_tickets 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f50237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 5 monclient: authenticate success, global_id 4352 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 10 monclient: _renew_subs 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c139740 con 0x7f503c13f710 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 10 monclient: _renew_subs 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c1db540 con 0x7f503c13f710 2024-03-25T17:42:03.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.518+0000 7f504165c640 1 librados: init done 2024-03-25T17:42:03.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.519+0000 7f504165c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:03.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.519+0000 7f504165c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:03.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.519+0000 7f504165c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:03.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.519+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c137c00 con 0x7f503c13f710 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.846+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5030001220 con 0x7f503c13f710 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.846+0000 7f50237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.846+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f5030004c30 con 0x7f503c13f710 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.846+0000 7f50237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.846+0000 7f50237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.846+0000 7f50237fe640 10 monclient: dump: 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.847 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:03.848 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:03.848 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:03.848 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:03.848 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:03.848 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:03.848 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:03.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.846+0000 7f50397fa640 4 set_mon_vals no callback set 2024-03-25T17:42:03.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.847+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f503001ac40 con 0x7f503c13f710 2024-03-25T17:42:03.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.848+0000 7f503affd640 10 monclient: get_auth_request con 0x7f502806df60 auth_method 0 2024-03-25T17:42:03.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.848+0000 7f50237fe640 10 client.4352.objecter ms_handle_connect 0x7f502806df60 2024-03-25T17:42:03.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.853+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f5030054840 con 0x7f503c13f710 2024-03-25T17:42:03.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.853+0000 7f50237fe640 10 client.4352.objecter ms_dispatch 0x7f503c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:42:03.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.853+0000 7f50237fe640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:42:03.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.853+0000 7f50237fe640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:42:03.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.853+0000 7f50237fe640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-25T17:42:03.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.853+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5030050180 con 0x7f503c13f710 2024-03-25T17:42:03.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.853+0000 7f50237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:03.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:03.853+0000 7f50237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:42:04.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.110+0000 7f504165c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:04.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.110+0000 7f504165c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:04.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.110+0000 7f504165c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:04.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.110+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f503c0627b0 con 0x7f503c13f710 2024-03-25T17:42:04.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.133+0000 7f50237fe640 1 -- 172.21.15.44:0/3267197077 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f5030052280 con 0x7f503c13f710 2024-03-25T17:42:04.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.133+0000 7f50237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:04.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.133+0000 7f50237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:42:04.134 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:42:04.134 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 10 librados: watch_flush enter 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 10 librados: watch_flush exit 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 10 client.4352.objecter successfully canceled tick 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 >> v1:172.21.15.44:6800/3257043132 conn(0x7f502806df60 legacy=0x7f5028070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 10 monclient: shutdown 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 >> v1:172.21.15.44:6790/0 conn(0x7f503c13f710 legacy=0x7f503c133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 shutdown_connections 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 >> 172.21.15.44:0/3267197077 conn(0x7f503c0b7ea0 msgr2=0x7f503c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 shutdown_connections 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 1 -- 172.21.15.44:0/3267197077 wait complete. 2024-03-25T17:42:04.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:04.136+0000 7f504165c640 1 librados: shutdown 2024-03-25T17:42:04.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:42:10.954 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:42:11.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.068+0000 7f9c0b606640 10 monclient: get_monmap_and_config 2024-03-25T17:42:11.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.068+0000 7f9c0b606640 10 monclient: build_initial_monmap 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.068+0000 7f9c0b606640 10 monclient: monmap: 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:42:11.070114+0000 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:42:11.070114+0000 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:42:11.070 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:11.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.068+0000 7f9c0b606640 1 Processor -- start 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.069+0000 7f9c0b606640 1 -- start start 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.069+0000 7f9c0b606640 10 monclient: init 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.069+0000 7f9c0b606640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.069+0000 7f9c0b606640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.069+0000 7f9c0b606640 10 monclient(hunting): picked mon.noname-a con 0x7f9c0413fc60 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.069+0000 7f9c0b606640 10 monclient(hunting): picked mon.noname-b con 0x7f9c04070570 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.069+0000 7f9c0b606640 10 monclient(hunting): picked mon.noname-c con 0x7f9c040727a0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.070+0000 7f9c0b606640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0413c9f0 con 0x7f9c0413fc60 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.070+0000 7f9c0b606640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0406fac0 con 0x7f9c040727a0 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.070+0000 7f9c0b606640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c040715d0 con 0x7f9c04070570 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.070+0000 7f9c0b606640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.070+0000 7f9c0b606640 10 monclient(hunting): authenticate will time out at 6134.298785s 2024-03-25T17:42:11.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.079+0000 7f9c08ba1640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f9c04070570 0x7f9c04072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:36838/0 (socket says 172.21.15.44:36838) 2024-03-25T17:42:11.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.079+0000 7f9c08ba1640 1 -- 172.21.15.44:0/2869898695 learned_addr learned my addr 172.21.15.44:0/2869898695 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:11.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.092+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639673432 0 0) 0x7f9c040715d0 con 0x7f9c04070570 2024-03-25T17:42:11.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.092+0000 7f9bfbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:42:11.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.092+0000 7f9bfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:11.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.092+0000 7f9bfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:11.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.092+0000 7f9bfbfff640 10 monclient(hunting): my global_id is 4355 2024-03-25T17:42:11.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.092+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf0003520 con 0x7f9c04070570 2024-03-25T17:42:11.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3959073023 0 0) 0x7f9bf0003520 con 0x7f9c04070570 2024-03-25T17:42:11.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 >> v1:172.21.15.111:6789/0 conn(0x7f9c040727a0 legacy=0x7f9c0414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:11.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 >> v1:172.21.15.44:6789/0 conn(0x7f9c0413fc60 legacy=0x7f9c0406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:11.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 10 monclient: _finish_hunting 0 2024-03-25T17:42:11.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 1 monclient: found mon.noname-b 2024-03-25T17:42:11.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:11.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c040718b0 con 0x7f9c04070570 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 10 monclient: _finish_auth 0 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 10 monclient: _check_auth_tickets 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9bf4001690 con 0x7f9c04070570 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9c0b606640 5 monclient: authenticate success, global_id 4355 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9c0b606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9bfbfff640 10 monclient: dump: 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:11.102 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:11.103 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:11.103 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:11.103 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:11.103 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:11.103 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:11.103 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:11.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.099+0000 7f9c0b606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.113+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bf40017f0 con 0x7f9c04070570 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.113+0000 7f9bfbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.113+0000 7f9bfbfff640 1 -- 172.21.15.44:0/2869898695 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9bf4001c00 con 0x7f9c04070570 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.113+0000 7f9c0b606640 10 monclient: get_monmap_and_config success 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.113+0000 7f9c0b606640 4 set_mon_vals no callback set 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.113+0000 7f9bfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.113+0000 7f9bfbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.113+0000 7f9bfbfff640 10 monclient: dump: 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:11.115 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:11.116 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:11.116 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:11.116 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:11.116 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.114+0000 7f9c0b606640 10 monclient: shutdown 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.114+0000 7f9c0b606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.114+0000 7f9c0b606640 1 -- 172.21.15.44:0/2869898695 >> v1:172.21.15.44:6790/0 conn(0x7f9c04070570 legacy=0x7f9c04072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.114+0000 7f9c0b606640 1 -- 172.21.15.44:0/2869898695 shutdown_connections 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.114+0000 7f9c0b606640 1 -- 172.21.15.44:0/2869898695 >> 172.21.15.44:0/2869898695 conn(0x7f9c040b4a80 msgr2=0x7f9c040b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.114+0000 7f9c0b606640 1 -- 172.21.15.44:0/2869898695 shutdown_connections 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.114+0000 7f9c0b606640 1 -- 172.21.15.44:0/2869898695 wait complete. 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 10 monclient: build_initial_monmap 2024-03-25T17:42:11.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 10 monclient: monmap: 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:11.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 1 librados: starting msgr at 2024-03-25T17:42:11.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 1 librados: starting objecter 2024-03-25T17:42:11.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 1 Processor -- start 2024-03-25T17:42:11.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 1 -- start start 2024-03-25T17:42:11.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 1 librados: setting wanted keys 2024-03-25T17:42:11.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 1 librados: calling monclient init 2024-03-25T17:42:11.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.115+0000 7f9c0b606640 10 monclient: init 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 10 monclient(hunting): picked mon.noname-b con 0x7f9c04070570 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 10 monclient(hunting): picked mon.noname-a con 0x7f9c040727a0 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 10 monclient(hunting): picked mon.noname-c con 0x7f9c0413fc60 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c04140aa0 con 0x7f9c040727a0 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c04145850 con 0x7f9c04070570 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0414b6b0 con 0x7f9c0413fc60 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:11.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.116+0000 7f9c0b606640 10 monclient(hunting): authenticate will time out at 6134.345559s 2024-03-25T17:42:11.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.121+0000 7f9c09ba3640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f9c0413fc60 0x7f9c041e2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:36892/0 (socket says 172.21.15.44:36892) 2024-03-25T17:42:11.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.121+0000 7f9c09ba3640 1 -- 172.21.15.44:0/3797826046 learned_addr learned my addr 172.21.15.44:0/3797826046 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:11.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.127+0000 7f9bf9ffb640 10 client.?.objecter ms_handle_connect 0x7f9c0413fc60 2024-03-25T17:42:11.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.127+0000 7f9bf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:11.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.131+0000 7f9bf9ffb640 10 client.?.objecter ms_handle_connect 0x7f9c040727a0 2024-03-25T17:42:11.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.131+0000 7f9bf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:11.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435795992 0 0) 0x7f9c0414b6b0 con 0x7f9c0413fc60 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 10 monclient(hunting): my global_id is 4358 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be8004ee0 con 0x7f9c0413fc60 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257614909 0 0) 0x7f9c04140aa0 con 0x7f9c040727a0 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 10 monclient(hunting): my global_id is 4371 2024-03-25T17:42:11.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.138+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c0414b6b0 con 0x7f9c040727a0 2024-03-25T17:42:11.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.140+0000 7f9bf9ffb640 10 client.?.objecter ms_handle_connect 0x7f9c04070570 2024-03-25T17:42:11.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.140+0000 7f9bf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:11.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.147+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 74638393 0 0) 0x7f9be8004ee0 con 0x7f9c0413fc60 2024-03-25T17:42:11.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.147+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c04140aa0 con 0x7f9c0413fc60 2024-03-25T17:42:11.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.147+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9bfc001c60 con 0x7f9c0413fc60 2024-03-25T17:42:11.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.147+0000 7f9bf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:11.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.147+0000 7f9bf9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:42:11.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.147+0000 7f9bf9ffb640 10 monclient(hunting): dump: 2024-03-25T17:42:11.148 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:11.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.147+0000 7f9bf9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:42:11.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.154+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3673488934 0 0) 0x7f9c04140aa0 con 0x7f9c0413fc60 2024-03-25T17:42:11.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.154+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 >> v1:172.21.15.111:6789/0 conn(0x7f9c04070570 legacy=0x7f9c041db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:11.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 >> v1:172.21.15.44:6789/0 conn(0x7f9c040727a0 legacy=0x7f9c04138c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 1 monclient: found mon.c 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c041447a0 con 0x7f9c0413fc60 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 10 monclient: _finish_auth 0 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9bf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 5 monclient: authenticate success, global_id 4358 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 10 monclient: _renew_subs 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c04144350 con 0x7f9c0413fc60 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:42:11.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 10 monclient: _renew_subs 2024-03-25T17:42:11.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:11.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c041db930 con 0x7f9c0413fc60 2024-03-25T17:42:11.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.155+0000 7f9c0b606640 1 librados: init done 2024-03-25T17:42:11.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.156+0000 7f9c0b606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:11.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.156+0000 7f9c0b606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:11.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.156+0000 7f9c0b606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:11.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.156+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c041449f0 con 0x7f9c0413fc60 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.180+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bfc003990 con 0x7f9c0413fc60 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.180+0000 7f9bf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.180+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9bfc003d60 con 0x7f9c0413fc60 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.180+0000 7f9bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.180+0000 7f9bf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.180+0000 7f9bf9ffb640 10 monclient: dump: 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:11.182 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:11.183 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:11.183 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:11.183 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:11.183 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:11.183 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:11.183 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:11.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.180+0000 7f9bfb7fe640 4 set_mon_vals no callback set 2024-03-25T17:42:11.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.181+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f9bfc003f80 con 0x7f9c0413fc60 2024-03-25T17:42:11.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.182+0000 7f9c093a2640 10 monclient: get_auth_request con 0x7f9be806dd40 auth_method 0 2024-03-25T17:42:11.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.182+0000 7f9bf9ffb640 10 client.4358.objecter ms_handle_connect 0x7f9be806dd40 2024-03-25T17:42:11.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.483+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f9bfc054b10 con 0x7f9c0413fc60 2024-03-25T17:42:11.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.483+0000 7f9bf9ffb640 10 client.4358.objecter ms_dispatch 0x7f9c04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:42:11.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.483+0000 7f9bf9ffb640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:42:11.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.483+0000 7f9bf9ffb640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:42:11.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.483+0000 7f9bf9ffb640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-25T17:42:11.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.483+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9bfc050620 con 0x7f9c0413fc60 2024-03-25T17:42:11.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.483+0000 7f9bf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:11.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.483+0000 7f9bf9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:42:11.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.744+0000 7f9c0b606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:11.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.744+0000 7f9c0b606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:11.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.744+0000 7f9c0b606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:11.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.744+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c041444e0 con 0x7f9c0413fc60 2024-03-25T17:42:11.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.775+0000 7f9bf9ffb640 1 -- 172.21.15.44:0/3797826046 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f9bfc052550 con 0x7f9c0413fc60 2024-03-25T17:42:11.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.775+0000 7f9bf9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:11.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.775+0000 7f9bf9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:42:11.777 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:42:11.777 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:42:11.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 10 librados: watch_flush enter 2024-03-25T17:42:11.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 10 librados: watch_flush exit 2024-03-25T17:42:11.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-25T17:42:11.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 10 client.4358.objecter successfully canceled tick 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 >> v1:172.21.15.44:6800/3257043132 conn(0x7f9be806dd40 legacy=0x7f9be8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 10 monclient: shutdown 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 >> v1:172.21.15.44:6790/0 conn(0x7f9c0413fc60 legacy=0x7f9c041e2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 shutdown_connections 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 >> 172.21.15.44:0/3797826046 conn(0x7f9c040b4a80 msgr2=0x7f9c040ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 shutdown_connections 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 1 -- 172.21.15.44:0/3797826046 wait complete. 2024-03-25T17:42:11.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:11.777+0000 7f9c0b606640 1 librados: shutdown 2024-03-25T17:42:11.790 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:42:17.606 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:34.961 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:17.623 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:34.979 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:17.662 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:35.018 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:17.669 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:34.989 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:17.674 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:34.994 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:17.680 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:35.036 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:17.692 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:35.048 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:17.699 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:35.019 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:17.726 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:35.082 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:18.597 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:42:18.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.691+0000 7f2d2e99d640 10 monclient: get_monmap_and_config 2024-03-25T17:42:18.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.691+0000 7f2d2e99d640 10 monclient: build_initial_monmap 2024-03-25T17:42:18.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.691+0000 7f2d2e99d640 10 monclient: monmap: 2024-03-25T17:42:18.692 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:42:18.692849+0000 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:42:18.692849+0000 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:42:18.693 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:18.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.692+0000 7f2d2e99d640 1 Processor -- start 2024-03-25T17:42:18.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.692+0000 7f2d2e99d640 1 -- start start 2024-03-25T17:42:18.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.692+0000 7f2d2e99d640 10 monclient: init 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d280b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d280b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d2814be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280af940 con 0x7f2d280b1730 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280b0a70 con 0x7f2d2814be90 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280b0d50 con 0x7f2d280b4d60 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:18.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.693+0000 7f2d2e99d640 10 monclient(hunting): authenticate will time out at 6141.921095s 2024-03-25T17:42:18.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.708+0000 7f2d277fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f2d280b1730 0x7f2d280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:59816/0 (socket says 172.21.15.44:59816) 2024-03-25T17:42:18.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.708+0000 7f2d277fe640 1 -- 172.21.15.44:0/2970895678 learned_addr learned my addr 172.21.15.44:0/2970895678 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:18.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.717+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726413311 0 0) 0x7f2d280af940 con 0x7f2d280b1730 2024-03-25T17:42:18.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.717+0000 7f2d26ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:42:18.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.717+0000 7f2d26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:18.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.717+0000 7f2d26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:18.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.717+0000 7f2d26ffd640 10 monclient(hunting): my global_id is 4374 2024-03-25T17:42:18.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.717+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d10003520 con 0x7f2d280b1730 2024-03-25T17:42:18.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922796464 0 0) 0x7f2d10003520 con 0x7f2d280b1730 2024-03-25T17:42:18.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 >> v1:172.21.15.44:6790/0 conn(0x7f2d280b4d60 legacy=0x7f2d2813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:42:18.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 >> v1:172.21.15.111:6789/0 conn(0x7f2d2814be90 legacy=0x7f2d2814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:42:18.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:42:18.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 1 monclient: found mon.noname-a 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d280b1030 con 0x7f2d280b1730 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 10 monclient: _finish_auth 0 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d2e99d640 5 monclient: authenticate success, global_id 4374 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2d1c001690 con 0x7f2d280b1730 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d2e99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d26ffd640 10 monclient: dump: 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:18.743 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:18.744 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:18.744 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:18.744 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:18.744 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:18.744 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:18.744 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:18.744 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:18.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.741+0000 7f2d2e99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:18.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.775+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d1c001860 con 0x7f2d280b1730 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.775+0000 7f2d26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.775+0000 7f2d26ffd640 1 -- 172.21.15.44:0/2970895678 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2d1c001cb0 con 0x7f2d280b1730 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.775+0000 7f2d26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.775+0000 7f2d26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.775+0000 7f2d26ffd640 10 monclient: dump: 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:18.776 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:18.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.775+0000 7f2d2e99d640 10 monclient: get_monmap_and_config success 2024-03-25T17:42:18.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.775+0000 7f2d2e99d640 4 set_mon_vals no callback set 2024-03-25T17:42:18.778 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.776+0000 7f2d2e99d640 10 monclient: shutdown 2024-03-25T17:42:18.778 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.776+0000 7f2d2e99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:18.778 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.776+0000 7f2d2e99d640 1 -- 172.21.15.44:0/2970895678 >> v1:172.21.15.44:6789/0 conn(0x7f2d280b1730 legacy=0x7f2d280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:18.778 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.776+0000 7f2d2e99d640 1 -- 172.21.15.44:0/2970895678 shutdown_connections 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.776+0000 7f2d2e99d640 1 -- 172.21.15.44:0/2970895678 >> 172.21.15.44:0/2970895678 conn(0x7f2d280617a0 msgr2=0x7f2d28061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.776+0000 7f2d2e99d640 1 -- 172.21.15.44:0/2970895678 shutdown_connections 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.776+0000 7f2d2e99d640 1 -- 172.21.15.44:0/2970895678 wait complete. 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.777+0000 7f2d2e99d640 10 monclient: build_initial_monmap 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.777+0000 7f2d2e99d640 10 monclient: monmap: 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:42:18.779 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:42:18.780 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:18.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.777+0000 7f2d2e99d640 1 librados: starting msgr at 2024-03-25T17:42:18.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.777+0000 7f2d2e99d640 1 librados: starting objecter 2024-03-25T17:42:18.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.777+0000 7f2d2e99d640 1 Processor -- start 2024-03-25T17:42:18.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 1 -- start start 2024-03-25T17:42:18.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 1 librados: setting wanted keys 2024-03-25T17:42:18.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 1 librados: calling monclient init 2024-03-25T17:42:18.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 10 monclient: init 2024-03-25T17:42:18.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:18.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:42:18.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d280b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:18.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d280b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:18.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d2814be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.778+0000 7f2d2e99d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280a9b80 con 0x7f2d2814be90 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.779+0000 7f2d2e99d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280aacd0 con 0x7f2d280b4d60 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.779+0000 7f2d2e99d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d281dafd0 con 0x7f2d280b1730 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.779+0000 7f2d2e99d640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.779+0000 7f2d2e99d640 10 monclient(hunting): authenticate will time out at 6142.006697s 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.779+0000 7f2d277fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2d280b4d60 0x7f2d281db4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:36030/0 (socket says 172.21.15.44:36030) 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.779+0000 7f2d277fe640 1 -- 172.21.15.44:0/1489890738 learned_addr learned my addr 172.21.15.44:0/1489890738 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.779+0000 7f2d24ff9640 10 client.?.objecter ms_handle_connect 0x7f2d280b4d60 2024-03-25T17:42:18.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.779+0000 7f2d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:18.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.795+0000 7f2d24ff9640 10 client.?.objecter ms_handle_connect 0x7f2d280b1730 2024-03-25T17:42:18.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.795+0000 7f2d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:18.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.841+0000 7f2d24ff9640 10 client.?.objecter ms_handle_connect 0x7f2d2814be90 2024-03-25T17:42:18.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.841+0000 7f2d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:18.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.848+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481362387 0 0) 0x7f2d280a9b80 con 0x7f2d2814be90 2024-03-25T17:42:18.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.848+0000 7f2d24ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:42:18.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.848+0000 7f2d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:18.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.848+0000 7f2d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:18.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.848+0000 7f2d24ff9640 10 monclient(hunting): my global_id is 4377 2024-03-25T17:42:18.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.848+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d14004eb0 con 0x7f2d2814be90 2024-03-25T17:42:18.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.855+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780016245 0 0) 0x7f2d280aacd0 con 0x7f2d280b4d60 2024-03-25T17:42:18.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.855+0000 7f2d24ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:42:18.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.855+0000 7f2d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:18.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.855+0000 7f2d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:18.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.855+0000 7f2d24ff9640 10 monclient(hunting): my global_id is 4363 2024-03-25T17:42:18.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.855+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d280a9b80 con 0x7f2d280b4d60 2024-03-25T17:42:18.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.858+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642052963 0 0) 0x7f2d281dafd0 con 0x7f2d280b1730 2024-03-25T17:42:18.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.858+0000 7f2d24ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:42:18.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.858+0000 7f2d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:18.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.858+0000 7f2d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:18.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.858+0000 7f2d24ff9640 10 monclient(hunting): my global_id is 4361 2024-03-25T17:42:18.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.858+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d280aacd0 con 0x7f2d280b1730 2024-03-25T17:42:18.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.878+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4126613569 0 0) 0x7f2d280aacd0 con 0x7f2d280b1730 2024-03-25T17:42:18.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.878+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d281dafd0 con 0x7f2d280b1730 2024-03-25T17:42:18.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.878+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2d180017d0 con 0x7f2d280b1730 2024-03-25T17:42:18.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.878+0000 7f2d24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:18.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.878+0000 7f2d24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:42:18.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.878+0000 7f2d24ff9640 10 monclient(hunting): dump: 2024-03-25T17:42:18.879 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:18.879 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:18.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.878+0000 7f2d24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.879+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3425355110 0 0) 0x7f2d280a9b80 con 0x7f2d280b4d60 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.879+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d280aacd0 con 0x7f2d280b4d60 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.879+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2d1c001b70 con 0x7f2d280b4d60 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.879+0000 7f2d24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.879+0000 7f2d24ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.879+0000 7f2d24ff9640 10 monclient(hunting): dump: 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:18.881 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:18.882 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:18.882 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:18.882 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:18.882 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:18.882 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:18.882 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:18.882 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:18.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 180534832 0 0) 0x7f2d281dafd0 con 0x7f2d280b1730 2024-03-25T17:42:18.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 >> v1:172.21.15.111:6789/0 conn(0x7f2d280b4d60 legacy=0x7f2d281db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:18.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 >> v1:172.21.15.44:6789/0 conn(0x7f2d2814be90 legacy=0x7f2d281e2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:18.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:42:18.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 1 monclient: found mon.c 2024-03-25T17:42:18.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:18.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28138d60 con 0x7f2d280b1730 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 10 monclient: _finish_auth 0 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 5 monclient: authenticate success, global_id 4361 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 10 monclient: _renew_subs 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d281dbbe0 con 0x7f2d280b1730 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 10 monclient: _renew_subs 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2813b7c0 con 0x7f2d280b1730 2024-03-25T17:42:18.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.894+0000 7f2d2e99d640 1 librados: init done 2024-03-25T17:42:18.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.895+0000 7f2d2e99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:18.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.895+0000 7f2d2e99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:18.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.895+0000 7f2d2e99d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:18.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.895+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d281401f0 con 0x7f2d280b1730 2024-03-25T17:42:18.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.902+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d18001cc0 con 0x7f2d280b1730 2024-03-25T17:42:18.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.902+0000 7f2d24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:18.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:18.902+0000 7f2d267fc640 4 set_mon_vals no callback set 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.233+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2d18003790 con 0x7f2d280b1730 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.233+0000 7f2d24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.233+0000 7f2d24ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.233+0000 7f2d24ff9640 10 monclient: dump: 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:19.234 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:19.235 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:19.235 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:19.235 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:19.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.234+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f2d18019760 con 0x7f2d280b1730 2024-03-25T17:42:19.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.234+0000 7f2d277fe640 10 monclient: get_auth_request con 0x7f2d1406df30 auth_method 0 2024-03-25T17:42:19.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.234+0000 7f2d24ff9640 10 client.4361.objecter ms_handle_connect 0x7f2d1406df30 2024-03-25T17:42:19.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.240+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f2d180534d0 con 0x7f2d280b1730 2024-03-25T17:42:19.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.240+0000 7f2d24ff9640 10 client.4361.objecter ms_dispatch 0x7f2d28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:42:19.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.240+0000 7f2d24ff9640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:42:19.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.240+0000 7f2d24ff9640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:42:19.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.240+0000 7f2d24ff9640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-25T17:42:19.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.240+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d1804efe0 con 0x7f2d280b1730 2024-03-25T17:42:19.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.240+0000 7f2d24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:19.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.240+0000 7f2d24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:42:19.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.497+0000 7f2d2e99d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:19.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.497+0000 7f2d2e99d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:19.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.497+0000 7f2d2e99d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:19.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.497+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d281db160 con 0x7f2d280b1730 2024-03-25T17:42:19.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.555+0000 7f2d24ff9640 1 -- 172.21.15.44:0/1489890738 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f2d18050f10 con 0x7f2d280b1730 2024-03-25T17:42:19.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.555+0000 7f2d24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:19.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.555+0000 7f2d24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:42:19.556 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:42:19.557 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:42:19.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 10 librados: watch_flush enter 2024-03-25T17:42:19.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 10 librados: watch_flush exit 2024-03-25T17:42:19.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 10 client.4361.objecter successfully canceled tick 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 >> v1:172.21.15.44:6800/3257043132 conn(0x7f2d1406df30 legacy=0x7f2d140703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 10 monclient: shutdown 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 >> v1:172.21.15.44:6790/0 conn(0x7f2d280b1730 legacy=0x7f2d280a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 shutdown_connections 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 >> 172.21.15.44:0/1489890738 conn(0x7f2d280617a0 msgr2=0x7f2d28144f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.558+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 shutdown_connections 2024-03-25T17:42:19.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.559+0000 7f2d2e99d640 1 -- 172.21.15.44:0/1489890738 wait complete. 2024-03-25T17:42:19.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:19.559+0000 7f2d2e99d640 1 librados: shutdown 2024-03-25T17:42:19.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:42:26.377 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:42:26.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.491+0000 7fca01df4640 10 monclient: get_monmap_and_config 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.491+0000 7fca01df4640 10 monclient: build_initial_monmap 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.491+0000 7fca01df4640 10 monclient: monmap: 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:42:26.492415+0000 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:42:26.492415+0000 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:42:26.492 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:26.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.492+0000 7fca01df4640 1 Processor -- start 2024-03-25T17:42:26.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.492+0000 7fca01df4640 1 -- start start 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.492+0000 7fca01df4640 10 monclient: init 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 10 monclient(hunting): picked mon.noname-c con 0x7fc9fc0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 10 monclient(hunting): picked mon.noname-a con 0x7fc9fc13f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 10 monclient(hunting): picked mon.noname-b con 0x7fc9fc1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc142520 con 0x7fc9fc13f710 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc0b28a0 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc0b2a30 con 0x7fc9fc1430c0 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:26.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.493+0000 7fca01df4640 10 monclient(hunting): authenticate will time out at 6149.721067s 2024-03-25T17:42:26.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.497+0000 7fc9faffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc9fc13f710 0x7fc9fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:50306/0 (socket says 172.21.15.44:50306) 2024-03-25T17:42:26.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.497+0000 7fc9faffd640 1 -- 172.21.15.44:0/3179315613 learned_addr learned my addr 172.21.15.44:0/3179315613 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:26.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.518+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670855173 0 0) 0x7fc9fc0b28a0 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.518+0000 7fc9fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:42:26.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.518+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:26.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.518+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:26.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.518+0000 7fc9fa7fc640 10 monclient(hunting): my global_id is 4366 2024-03-25T17:42:26.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.518+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e4003520 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.525+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522606575 0 0) 0x7fc9fc142520 con 0x7fc9fc13f710 2024-03-25T17:42:26.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.525+0000 7fc9fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:42:26.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.525+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:26.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.525+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:26.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.525+0000 7fc9fa7fc640 10 monclient(hunting): my global_id is 4380 2024-03-25T17:42:26.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.525+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9fc0b28a0 con 0x7fc9fc13f710 2024-03-25T17:42:26.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.527+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771218162 0 0) 0x7fc9fc0b2a30 con 0x7fc9fc1430c0 2024-03-25T17:42:26.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.527+0000 7fc9fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:42:26.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.527+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:26.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.527+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:26.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.527+0000 7fc9fa7fc640 10 monclient(hunting): my global_id is 4364 2024-03-25T17:42:26.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.527+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9fc142520 con 0x7fc9fc1430c0 2024-03-25T17:42:26.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.530+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2851039147 0 0) 0x7fc9e4003520 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.530+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 >> v1:172.21.15.44:6790/0 conn(0x7fc9fc1430c0 legacy=0x7fc9fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:26.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.530+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 >> v1:172.21.15.44:6789/0 conn(0x7fc9fc13f710 legacy=0x7fc9fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:26.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.530+0000 7fc9fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:42:26.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.530+0000 7fc9fa7fc640 1 monclient: found mon.noname-c 2024-03-25T17:42:26.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc0b44c0 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 10 monclient: _finish_auth 0 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc9ec001690 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fca01df4640 5 monclient: authenticate success, global_id 4366 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fca01df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fc9fa7fc640 10 monclient: dump: 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:26.532 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:26.533 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.531+0000 7fca01df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:26.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.534+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9ec0017f0 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.534+0000 7fc9fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:26.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.534+0000 7fc9fa7fc640 1 -- 172.21.15.44:0/3179315613 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc9ec001c00 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.534+0000 7fca01df4640 10 monclient: get_monmap_and_config success 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.534+0000 7fca01df4640 4 set_mon_vals no callback set 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.534+0000 7fc9fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.534+0000 7fc9fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.534+0000 7fc9fa7fc640 10 monclient: dump: 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:26.535 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.535+0000 7fca01df4640 10 monclient: shutdown 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.535+0000 7fca01df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.535+0000 7fca01df4640 1 -- 172.21.15.44:0/3179315613 >> v1:172.21.15.111:6789/0 conn(0x7fc9fc0a98f0 legacy=0x7fc9fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.535+0000 7fca01df4640 1 -- 172.21.15.44:0/3179315613 shutdown_connections 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.535+0000 7fca01df4640 1 -- 172.21.15.44:0/3179315613 >> 172.21.15.44:0/3179315613 conn(0x7fc9fc0b7ea0 msgr2=0x7fc9fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.535+0000 7fca01df4640 1 -- 172.21.15.44:0/3179315613 shutdown_connections 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.535+0000 7fca01df4640 1 -- 172.21.15.44:0/3179315613 wait complete. 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.536+0000 7fca01df4640 10 monclient: build_initial_monmap 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.536+0000 7fca01df4640 10 monclient: monmap: 2024-03-25T17:42:26.537 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.536+0000 7fca01df4640 1 librados: starting msgr at 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.536+0000 7fca01df4640 1 librados: starting objecter 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.536+0000 7fca01df4640 1 Processor -- start 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.537+0000 7fca01df4640 1 -- start start 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.537+0000 7fca01df4640 1 librados: setting wanted keys 2024-03-25T17:42:26.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.537+0000 7fca01df4640 1 librados: calling monclient init 2024-03-25T17:42:26.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.537+0000 7fca01df4640 10 monclient: init 2024-03-25T17:42:26.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.537+0000 7fca01df4640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.537+0000 7fca01df4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.538+0000 7fca01df4640 10 monclient(hunting): picked mon.noname-a con 0x7fc9fc0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.538+0000 7fca01df4640 10 monclient(hunting): picked mon.noname-b con 0x7fc9fc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.538+0000 7fca01df4640 10 monclient(hunting): picked mon.noname-c con 0x7fc9fc1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.538+0000 7fca01df4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc138a50 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.538+0000 7fca01df4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc0ab9e0 con 0x7fc9fc13f710 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.538+0000 7fca01df4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc1d7880 con 0x7fc9fc1430c0 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.538+0000 7fca01df4640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:26.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.538+0000 7fca01df4640 10 monclient(hunting): authenticate will time out at 6149.765840s 2024-03-25T17:42:26.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.543+0000 7fc9fbfff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fc9fc1430c0 0x7fc9fc1e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:38538/0 (socket says 172.21.15.44:38538) 2024-03-25T17:42:26.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.543+0000 7fc9fbfff640 1 -- 172.21.15.44:0/3359762539 learned_addr learned my addr 172.21.15.44:0/3359762539 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:26.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.545+0000 7fc9e3fff640 10 client.?.objecter ms_handle_connect 0x7fc9fc1430c0 2024-03-25T17:42:26.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.545+0000 7fc9e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:26.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.548+0000 7fc9e3fff640 10 client.?.objecter ms_handle_connect 0x7fc9fc0a98f0 2024-03-25T17:42:26.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.548+0000 7fc9e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:26.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.550+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777975008 0 0) 0x7fc9fc1d7880 con 0x7fc9fc1430c0 2024-03-25T17:42:26.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.550+0000 7fc9e3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:42:26.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.550+0000 7fc9e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:26.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.550+0000 7fc9e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:26.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.550+0000 7fc9e3fff640 10 monclient(hunting): my global_id is 4367 2024-03-25T17:42:26.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.550+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e8004eb0 con 0x7fc9fc1430c0 2024-03-25T17:42:26.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3259207231 0 0) 0x7fc9e8004eb0 con 0x7fc9fc1430c0 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9fc1d7880 con 0x7fc9fc1430c0 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc9f4001c70 con 0x7fc9fc1430c0 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 10 monclient(hunting): dump: 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:26.563 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:26.564 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:26.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:42:26.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 10 client.?.objecter ms_handle_connect 0x7fc9fc13f710 2024-03-25T17:42:26.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.562+0000 7fc9e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:26.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.564+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398519802 0 0) 0x7fc9fc138a50 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.564+0000 7fc9e3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:42:26.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.564+0000 7fc9e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:26.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.564+0000 7fc9e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:26.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.564+0000 7fc9e3fff640 10 monclient(hunting): my global_id is 4383 2024-03-25T17:42:26.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.564+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e8004eb0 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.571+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4194101493 0 0) 0x7fc9e8004eb0 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.571+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9fc138a50 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.571+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc9ec001b00 con 0x7fc9fc0a98f0 2024-03-25T17:42:26.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.571+0000 7fc9e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.571+0000 7fc9e3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.571+0000 7fc9e3fff640 10 monclient(hunting): dump: 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:26.572 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.574+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3799081870 0 0) 0x7fc9fc1d7880 con 0x7fc9fc1430c0 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 >> v1:172.21.15.111:6789/0 conn(0x7fc9fc13f710 legacy=0x7fc9fc1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 >> v1:172.21.15.44:6789/0 conn(0x7fc9fc0a98f0 legacy=0x7fc9fc138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 1 monclient: found mon.c 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc1d84b0 con 0x7fc9fc1430c0 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 10 monclient: _finish_auth 0 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:42:26.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fc9e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 5 monclient: authenticate success, global_id 4367 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 10 monclient: _renew_subs 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9fc1d51f0 con 0x7fc9fc1430c0 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 10 monclient: _renew_subs 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9fc138e60 con 0x7fc9fc1430c0 2024-03-25T17:42:26.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.575+0000 7fca01df4640 1 librados: init done 2024-03-25T17:42:26.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.576+0000 7fca01df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:26.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.576+0000 7fca01df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:26.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.576+0000 7fca01df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:26.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.576+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9fc1d76e0 con 0x7fc9fc1430c0 2024-03-25T17:42:26.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.887+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9f4003990 con 0x7fc9fc1430c0 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.887+0000 7fc9e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.887+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc9f4001dd0 con 0x7fc9fc1430c0 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.887+0000 7fc9e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.887+0000 7fc9e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.887+0000 7fc9e3fff640 10 monclient: dump: 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:26.888 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:26.889 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:26.889 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:26.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.887+0000 7fc9f9ffb640 4 set_mon_vals no callback set 2024-03-25T17:42:26.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.888+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fc9f4003cc0 con 0x7fc9fc1430c0 2024-03-25T17:42:26.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.888+0000 7fc9fb7fe640 10 monclient: get_auth_request con 0x7fc9e806ded0 auth_method 0 2024-03-25T17:42:26.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.889+0000 7fc9e3fff640 10 client.4367.objecter ms_handle_connect 0x7fc9e806ded0 2024-03-25T17:42:26.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.894+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fc9f4054970 con 0x7fc9fc1430c0 2024-03-25T17:42:26.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.894+0000 7fc9e3fff640 10 client.4367.objecter ms_dispatch 0x7fc9fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:42:26.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.894+0000 7fc9e3fff640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:42:26.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.894+0000 7fc9e3fff640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:42:26.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.894+0000 7fc9e3fff640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-25T17:42:26.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.894+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9f40502b0 con 0x7fc9fc1430c0 2024-03-25T17:42:26.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.894+0000 7fc9e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:26.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:26.894+0000 7fc9e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:42:27.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.116+0000 7fca01df4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:27.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.116+0000 7fca01df4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:27.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.116+0000 7fca01df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:27.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.116+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9fc0627b0 con 0x7fc9fc1430c0 2024-03-25T17:42:27.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.166+0000 7fc9e3fff640 1 -- 172.21.15.44:0/3359762539 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fc9f40523b0 con 0x7fc9fc1430c0 2024-03-25T17:42:27.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.166+0000 7fc9e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:27.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.166+0000 7fc9e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:42:27.167 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:42:27.167 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:42:27.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.168+0000 7fca01df4640 10 librados: watch_flush enter 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.168+0000 7fca01df4640 10 librados: watch_flush exit 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.168+0000 7fca01df4640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.168+0000 7fca01df4640 10 client.4367.objecter successfully canceled tick 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.168+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 >> v1:172.21.15.44:6800/3257043132 conn(0x7fc9e806ded0 legacy=0x7fc9e8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.168+0000 7fca01df4640 10 monclient: shutdown 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.168+0000 7fca01df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.168+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 >> v1:172.21.15.44:6790/0 conn(0x7fc9fc1430c0 legacy=0x7fc9fc1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.169+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 shutdown_connections 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.169+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 >> 172.21.15.44:0/3359762539 conn(0x7fc9fc0b7ea0 msgr2=0x7fc9fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.169+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 shutdown_connections 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.169+0000 7fca01df4640 1 -- 172.21.15.44:0/3359762539 wait complete. 2024-03-25T17:42:27.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:27.169+0000 7fca01df4640 1 librados: shutdown 2024-03-25T17:42:27.181 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:42:33.988 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.102+0000 7f8e1d025640 10 monclient: get_monmap_and_config 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.102+0000 7f8e1d025640 10 monclient: build_initial_monmap 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.102+0000 7f8e1d025640 10 monclient: monmap: 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:42:34.103708+0000 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:42:34.103708+0000 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:34.103 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:42:34.104 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:42:34.104 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:34.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.103+0000 7f8e1d025640 1 Processor -- start 2024-03-25T17:42:34.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.103+0000 7f8e1d025640 1 -- start start 2024-03-25T17:42:34.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.103+0000 7f8e1d025640 10 monclient: init 2024-03-25T17:42:34.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.103+0000 7f8e1d025640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:34.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.103+0000 7f8e1d025640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:42:34.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.103+0000 7f8e1d025640 10 monclient(hunting): picked mon.noname-c con 0x7f8e180b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:34.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.103+0000 7f8e1d025640 10 monclient(hunting): picked mon.noname-b con 0x7f8e180b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:34.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.104+0000 7f8e1d025640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1814be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:34.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.104+0000 7f8e1d025640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e180af940 con 0x7f8e1814be90 2024-03-25T17:42:34.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.104+0000 7f8e1d025640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e180b0a70 con 0x7f8e180b4d60 2024-03-25T17:42:34.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.104+0000 7f8e1d025640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e180b0d50 con 0x7f8e180b1730 2024-03-25T17:42:34.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.104+0000 7f8e1d025640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:34.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.104+0000 7f8e1d025640 10 monclient(hunting): authenticate will time out at 6157.331737s 2024-03-25T17:42:34.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.104+0000 7f8e1659c640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f8e180b1730 0x7f8e180b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:57496/0 (socket says 172.21.15.44:57496) 2024-03-25T17:42:34.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.104+0000 7f8e1659c640 1 -- 172.21.15.44:0/2236199450 learned_addr learned my addr 172.21.15.44:0/2236199450 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:34.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.174+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856217593 0 0) 0x7f8e180af940 con 0x7f8e1814be90 2024-03-25T17:42:34.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.174+0000 7f8e15d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:42:34.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.174+0000 7f8e15d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:34.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.174+0000 7f8e15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:34.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.174+0000 7f8e15d9b640 10 monclient(hunting): my global_id is 4386 2024-03-25T17:42:34.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.174+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e00003520 con 0x7f8e1814be90 2024-03-25T17:42:34.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.188+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1819355444 0 0) 0x7f8e00003520 con 0x7f8e1814be90 2024-03-25T17:42:34.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 >> v1:172.21.15.44:6790/0 conn(0x7f8e180b1730 legacy=0x7f8e180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:34.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 >> v1:172.21.15.111:6789/0 conn(0x7f8e180b4d60 legacy=0x7f8e1813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 1 monclient: found mon.noname-a 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e180b1030 con 0x7f8e1814be90 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 10 monclient: _finish_auth 0 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e10001690 con 0x7f8e1814be90 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e1d025640 5 monclient: authenticate success, global_id 4386 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e1d025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e15d9b640 10 monclient: dump: 2024-03-25T17:42:34.190 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:34.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.189+0000 7f8e1d025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:34.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e100017f0 con 0x7f8e1814be90 2024-03-25T17:42:34.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e15d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:34.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e15d9b640 1 -- 172.21.15.44:0/2236199450 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e10001c00 con 0x7f8e1814be90 2024-03-25T17:42:34.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:34.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e15d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:42:34.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e15d9b640 10 monclient: dump: 2024-03-25T17:42:34.212 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:34.212 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e1d025640 10 monclient: get_monmap_and_config success 2024-03-25T17:42:34.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e1d025640 4 set_mon_vals no callback set 2024-03-25T17:42:34.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e1d025640 10 monclient: shutdown 2024-03-25T17:42:34.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e1d025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e1d025640 1 -- 172.21.15.44:0/2236199450 >> v1:172.21.15.44:6789/0 conn(0x7f8e1814be90 legacy=0x7f8e1814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e1d025640 1 -- 172.21.15.44:0/2236199450 shutdown_connections 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.212+0000 7f8e1d025640 1 -- 172.21.15.44:0/2236199450 >> 172.21.15.44:0/2236199450 conn(0x7f8e180617a0 msgr2=0x7f8e18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.213+0000 7f8e1d025640 1 -- 172.21.15.44:0/2236199450 shutdown_connections 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.213+0000 7f8e1d025640 1 -- 172.21.15.44:0/2236199450 wait complete. 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.213+0000 7f8e1d025640 10 monclient: build_initial_monmap 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.213+0000 7f8e1d025640 10 monclient: monmap: 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:42:34.215 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.214+0000 7f8e1d025640 1 librados: starting msgr at 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.214+0000 7f8e1d025640 1 librados: starting objecter 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.214+0000 7f8e1d025640 1 Processor -- start 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.214+0000 7f8e1d025640 1 -- start start 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.214+0000 7f8e1d025640 1 librados: setting wanted keys 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.214+0000 7f8e1d025640 1 librados: calling monclient init 2024-03-25T17:42:34.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.214+0000 7f8e1d025640 10 monclient: init 2024-03-25T17:42:34.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:34.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:42:34.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 10 monclient(hunting): picked mon.noname-c con 0x7f8e180b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:34.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 10 monclient(hunting): picked mon.noname-a con 0x7f8e180b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:34.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1814be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:34.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e181307c0 con 0x7f8e180b4d60 2024-03-25T17:42:34.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e18138f90 con 0x7f8e1814be90 2024-03-25T17:42:34.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e18132ab0 con 0x7f8e180b1730 2024-03-25T17:42:34.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:34.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.215+0000 7f8e1d025640 10 monclient(hunting): authenticate will time out at 6157.442894s 2024-03-25T17:42:34.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.229+0000 7f8e16d9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f8e180b1730 0x7f8e181393b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:57542/0 (socket says 172.21.15.44:57542) 2024-03-25T17:42:34.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.229+0000 7f8e16d9d640 1 -- 172.21.15.44:0/1380381292 learned_addr learned my addr 172.21.15.44:0/1380381292 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:34.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.231+0000 7f8dff7fe640 10 client.?.objecter ms_handle_connect 0x7f8e180b1730 2024-03-25T17:42:34.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.231+0000 7f8dff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:34.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.233+0000 7f8dff7fe640 10 client.?.objecter ms_handle_connect 0x7f8e180b4d60 2024-03-25T17:42:34.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.233+0000 7f8dff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:34.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.246+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264399210 0 0) 0x7f8e181307c0 con 0x7f8e180b4d60 2024-03-25T17:42:34.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.246+0000 7f8dff7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:42:34.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.246+0000 7f8dff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:34.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.246+0000 7f8dff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:34.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.246+0000 7f8dff7fe640 10 monclient(hunting): my global_id is 4389 2024-03-25T17:42:34.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.246+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e04004eb0 con 0x7f8e180b4d60 2024-03-25T17:42:34.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.247+0000 7f8dff7fe640 10 client.?.objecter ms_handle_connect 0x7f8e1814be90 2024-03-25T17:42:34.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.247+0000 7f8dff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:34.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.247+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249384164 0 0) 0x7f8e18132ab0 con 0x7f8e180b1730 2024-03-25T17:42:34.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.247+0000 7f8dff7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:42:34.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.247+0000 7f8dff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:34.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.247+0000 7f8dff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:34.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.247+0000 7f8dff7fe640 10 monclient(hunting): my global_id is 4373 2024-03-25T17:42:34.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.247+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e181307c0 con 0x7f8e180b1730 2024-03-25T17:42:34.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.257+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855302861 0 0) 0x7f8e18138f90 con 0x7f8e1814be90 2024-03-25T17:42:34.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.257+0000 7f8dff7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:42:34.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.257+0000 7f8dff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:34.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.257+0000 7f8dff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:34.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.257+0000 7f8dff7fe640 10 monclient(hunting): my global_id is 4375 2024-03-25T17:42:34.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.257+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e18132ab0 con 0x7f8e1814be90 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.259+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2722584744 0 0) 0x7f8e04004eb0 con 0x7f8e180b4d60 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.259+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e18138f90 con 0x7f8e180b4d60 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.259+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e0c0018a0 con 0x7f8e180b4d60 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.259+0000 7f8dff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.259+0000 7f8dff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.259+0000 7f8dff7fe640 10 monclient(hunting): dump: 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:34.260 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:34.261 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:34.261 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:34.261 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:34.261 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:34.261 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:34.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.259+0000 7f8dff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:42:34.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3873001908 0 0) 0x7f8e18138f90 con 0x7f8e180b4d60 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 >> v1:172.21.15.44:6790/0 conn(0x7f8e180b1730 legacy=0x7f8e181393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 >> v1:172.21.15.111:6789/0 conn(0x7f8e1814be90 legacy=0x7f8e181300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 1 monclient: found mon.a 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e18133a40 con 0x7f8e180b4d60 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 10 monclient: _finish_auth 0 2024-03-25T17:42:34.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8dff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 5 monclient: authenticate success, global_id 4389 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 10 monclient: _renew_subs 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e181316c0 con 0x7f8e180b4d60 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 10 monclient: _renew_subs 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e18137860 con 0x7f8e180b4d60 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.272+0000 7f8e1d025640 1 librados: init done 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.273+0000 7f8e1d025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.273+0000 7f8e1d025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.273+0000 7f8e1d025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:42:34.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.273+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e18132c40 con 0x7f8e180b4d60 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.594+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e0c001d10 con 0x7f8e180b4d60 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.594+0000 7f8dff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.594+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e0c004a40 con 0x7f8e180b4d60 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.594+0000 7f8dff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.594+0000 7f8dff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.594+0000 7f8dff7fe640 10 monclient: dump: 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:34.595 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:34.596 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:34.596 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:34.596 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:34.596 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:34.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.594+0000 7f8e1559a640 4 set_mon_vals no callback set 2024-03-25T17:42:34.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.594+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f8e0c004c60 con 0x7f8e180b4d60 2024-03-25T17:42:34.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.595+0000 7f8e16d9d640 10 monclient: get_auth_request con 0x7f8e0406ddf0 auth_method 0 2024-03-25T17:42:34.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.595+0000 7f8dff7fe640 10 client.4389.objecter ms_handle_connect 0x7f8e0406ddf0 2024-03-25T17:42:34.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.602+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f8e0c0547b0 con 0x7f8e180b4d60 2024-03-25T17:42:34.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.602+0000 7f8dff7fe640 10 client.4389.objecter ms_dispatch 0x7f8e18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:42:34.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.602+0000 7f8dff7fe640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:42:34.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.602+0000 7f8dff7fe640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:42:34.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.602+0000 7f8dff7fe640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-25T17:42:34.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.602+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e0c0502c0 con 0x7f8e180b4d60 2024-03-25T17:42:34.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.602+0000 7f8dff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:34.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.602+0000 7f8dff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:42:34.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.852+0000 7f8e1d025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:34.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.852+0000 7f8e1d025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:34.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.852+0000 7f8e1d025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:42:34.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.852+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e180627b0 con 0x7f8e180b4d60 2024-03-25T17:42:34.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.891+0000 7f8dff7fe640 1 -- 172.21.15.44:0/1380381292 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f8e0c0521f0 con 0x7f8e180b4d60 2024-03-25T17:42:34.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.891+0000 7f8dff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:34.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.891+0000 7f8dff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:42:34.892 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:42:34.892 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.893+0000 7f8e1d025640 10 librados: watch_flush enter 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 10 librados: watch_flush exit 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 10 client.4389.objecter successfully canceled tick 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 >> v1:172.21.15.44:6800/3257043132 conn(0x7f8e0406ddf0 legacy=0x7f8e040702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 10 monclient: shutdown 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 >> v1:172.21.15.44:6789/0 conn(0x7f8e180b4d60 legacy=0x7f8e1812f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:34.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 shutdown_connections 2024-03-25T17:42:34.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 >> 172.21.15.44:0/1380381292 conn(0x7f8e180617a0 msgr2=0x7f8e1813c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:34.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 shutdown_connections 2024-03-25T17:42:34.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 1 -- 172.21.15.44:0/1380381292 wait complete. 2024-03-25T17:42:34.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:34.894+0000 7f8e1d025640 1 librados: shutdown 2024-03-25T17:42:34.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:42:37.682 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:55.038 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:37.693 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:55.048 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:37.733 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:55.053 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:37.738 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:55.094 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:37.739 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:55.059 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:37.747 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:55.103 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:37.752 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:55.108 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:37.773 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:55.093 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:37.784 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:55.140 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:42:41.712 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:42:41.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.833+0000 7fc80369a640 10 monclient: get_monmap_and_config 2024-03-25T17:42:41.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.833+0000 7fc80369a640 10 monclient: build_initial_monmap 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.833+0000 7fc80369a640 10 monclient: monmap: 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:42:41.834334+0000 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:42:41.834334+0000 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:42:41.834 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.834+0000 7fc80369a640 1 Processor -- start 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.834+0000 7fc80369a640 1 -- start start 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.834+0000 7fc80369a640 10 monclient: init 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.834+0000 7fc80369a640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.834+0000 7fc80369a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.835+0000 7fc80369a640 10 monclient(hunting): picked mon.noname-b con 0x7fc7fc06c380 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.835+0000 7fc80369a640 10 monclient(hunting): picked mon.noname-c con 0x7fc7fc0b12d0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.835+0000 7fc80369a640 10 monclient(hunting): picked mon.noname-a con 0x7fc7fc0aedf0 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.835+0000 7fc80369a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc0b16f0 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.835+0000 7fc80369a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc0adfe0 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.835+0000 7fc80369a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc0add00 con 0x7fc7fc06c380 2024-03-25T17:42:41.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.835+0000 7fc80369a640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:41.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.835+0000 7fc80369a640 10 monclient(hunting): authenticate will time out at 6165.062787s 2024-03-25T17:42:41.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.843+0000 7fc800c35640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc7fc0b12d0 0x7fc7fc0ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:44166/0 (socket says 172.21.15.44:44166) 2024-03-25T17:42:41.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.843+0000 7fc800c35640 1 -- 172.21.15.44:0/1829306953 learned_addr learned my addr 172.21.15.44:0/1829306953 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:41.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.854+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026655663 0 0) 0x7fc7fc0b16f0 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.854+0000 7fc7f3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:42:41.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.854+0000 7fc7f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:41.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.854+0000 7fc7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:41.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.854+0000 7fc7f3fff640 10 monclient(hunting): my global_id is 4392 2024-03-25T17:42:41.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.854+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e8003520 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.863+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793996045 0 0) 0x7fc7fc0add00 con 0x7fc7fc06c380 2024-03-25T17:42:41.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.863+0000 7fc7f3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:42:41.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.863+0000 7fc7f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:41.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.863+0000 7fc7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:41.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.863+0000 7fc7f3fff640 10 monclient(hunting): my global_id is 4376 2024-03-25T17:42:41.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.863+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7fc0b16f0 con 0x7fc7fc06c380 2024-03-25T17:42:41.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.868+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239717425 0 0) 0x7fc7fc0adfe0 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.868+0000 7fc7f3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:42:41.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.868+0000 7fc7f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:41.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.868+0000 7fc7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:41.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.868+0000 7fc7f3fff640 10 monclient(hunting): my global_id is 4378 2024-03-25T17:42:41.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.868+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7fc0add00 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369464501 0 0) 0x7fc7fc0b16f0 con 0x7fc7fc06c380 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 >> v1:172.21.15.111:6789/0 conn(0x7fc7fc0b12d0 legacy=0x7fc7fc0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 >> v1:172.21.15.44:6789/0 conn(0x7fc7fc0aedf0 legacy=0x7fc7fc14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 1 monclient: found mon.noname-b 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc0ae2c0 con 0x7fc7fc06c380 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 10 monclient: _finish_auth 0 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc7f8001670 con 0x7fc7fc06c380 2024-03-25T17:42:41.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc7f3fff640 10 monclient: dump: 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc80369a640 5 monclient: authenticate success, global_id 4376 2024-03-25T17:42:41.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.882+0000 7fc80369a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:41.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.898+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7f80017d0 con 0x7fc7fc06c380 2024-03-25T17:42:41.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.898+0000 7fc7f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:41.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.898+0000 7fc7f3fff640 1 -- 172.21.15.44:0/1829306953 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc7f8001b60 con 0x7fc7fc06c380 2024-03-25T17:42:41.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.898+0000 7fc7f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.898+0000 7fc7f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.898+0000 7fc7f3fff640 10 monclient: dump: 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.898+0000 7fc80369a640 10 monclient: get_monmap_and_config success 2024-03-25T17:42:41.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.898+0000 7fc80369a640 4 set_mon_vals no callback set 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.899+0000 7fc80369a640 10 monclient: shutdown 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.899+0000 7fc80369a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.899+0000 7fc80369a640 1 -- 172.21.15.44:0/1829306953 >> v1:172.21.15.44:6790/0 conn(0x7fc7fc06c380 legacy=0x7fc7fc0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.899+0000 7fc80369a640 1 -- 172.21.15.44:0/1829306953 shutdown_connections 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.899+0000 7fc80369a640 1 -- 172.21.15.44:0/1829306953 >> 172.21.15.44:0/1829306953 conn(0x7fc7fc0b5d10 msgr2=0x7fc7fc0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.899+0000 7fc80369a640 1 -- 172.21.15.44:0/1829306953 shutdown_connections 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.899+0000 7fc80369a640 1 -- 172.21.15.44:0/1829306953 wait complete. 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.900+0000 7fc80369a640 10 monclient: build_initial_monmap 2024-03-25T17:42:41.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.900+0000 7fc80369a640 10 monclient: monmap: 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.900+0000 7fc80369a640 1 librados: starting msgr at 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.900+0000 7fc80369a640 1 librados: starting objecter 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.900+0000 7fc80369a640 1 Processor -- start 2024-03-25T17:42:41.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 1 -- start start 2024-03-25T17:42:41.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 1 librados: setting wanted keys 2024-03-25T17:42:41.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 1 librados: calling monclient init 2024-03-25T17:42:41.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 10 monclient: init 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 10 monclient(hunting): picked mon.noname-b con 0x7fc7fc06c380 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 10 monclient(hunting): picked mon.noname-a con 0x7fc7fc0aedf0 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 10 monclient(hunting): picked mon.noname-c con 0x7fc7fc0b12d0 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc06c8e0 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc1c34c0 con 0x7fc7fc06c380 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc1c30e0 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:41.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.901+0000 7fc80369a640 10 monclient(hunting): authenticate will time out at 6165.129414s 2024-03-25T17:42:41.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.913+0000 7fc800c35640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc7fc0aedf0 0x7fc7fc1c0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60962/0 (socket says 172.21.15.44:60962) 2024-03-25T17:42:41.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.913+0000 7fc800c35640 1 -- 172.21.15.44:0/223271441 learned_addr learned my addr 172.21.15.44:0/223271441 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:41.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.915+0000 7fc7f1ffb640 10 client.?.objecter ms_handle_connect 0x7fc7fc0aedf0 2024-03-25T17:42:41.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.915+0000 7fc7f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:41.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.921+0000 7fc7f1ffb640 10 client.?.objecter ms_handle_connect 0x7fc7fc0b12d0 2024-03-25T17:42:41.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.921+0000 7fc7f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:41.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.922+0000 7fc7f1ffb640 10 client.?.objecter ms_handle_connect 0x7fc7fc06c380 2024-03-25T17:42:41.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.922+0000 7fc7f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:41.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.924+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182208634 0 0) 0x7fc7fc06c8e0 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.924+0000 7fc7f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:42:41.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.924+0000 7fc7f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:41.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.924+0000 7fc7f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:41.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.924+0000 7fc7f1ffb640 10 monclient(hunting): my global_id is 4395 2024-03-25T17:42:41.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.924+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e0004eb0 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.938+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883149659 0 0) 0x7fc7fc1c34c0 con 0x7fc7fc06c380 2024-03-25T17:42:41.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.938+0000 7fc7f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:42:41.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.938+0000 7fc7f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:41.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.938+0000 7fc7f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:41.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.938+0000 7fc7f1ffb640 10 monclient(hunting): my global_id is 4381 2024-03-25T17:42:41.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.938+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7fc06c8e0 con 0x7fc7fc06c380 2024-03-25T17:42:41.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.950+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412699775 0 0) 0x7fc7fc1c30e0 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.950+0000 7fc7f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:42:41.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.950+0000 7fc7f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:41.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.950+0000 7fc7f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:41.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.950+0000 7fc7f1ffb640 10 monclient(hunting): my global_id is 4379 2024-03-25T17:42:41.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.950+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7fc1c34c0 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.962+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3119324388 0 0) 0x7fc7e0004eb0 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.962+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7fc1c30e0 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.962+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc7ec001870 con 0x7fc7fc0aedf0 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.962+0000 7fc7f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.962+0000 7fc7f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.962+0000 7fc7f1ffb640 10 monclient(hunting): dump: 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:41.963 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:41.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.962+0000 7fc7f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.964+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 200418844 0 0) 0x7fc7fc06c8e0 con 0x7fc7fc06c380 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.964+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7e0004eb0 con 0x7fc7fc06c380 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.964+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc7f8001ae0 con 0x7fc7fc06c380 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.964+0000 7fc7f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.964+0000 7fc7f1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.964+0000 7fc7f1ffb640 10 monclient(hunting): dump: 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:41.965 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:41.966 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:41.966 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:41.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.975+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1738870060 0 0) 0x7fc7fc1c34c0 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.975+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7fc06c8e0 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.975+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc7f4001cd0 con 0x7fc7fc0b12d0 2024-03-25T17:42:41.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.975+0000 7fc7f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:41.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.975+0000 7fc7f1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.975+0000 7fc7f1ffb640 10 monclient(hunting): dump: 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:41.976 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3328394052 0 0) 0x7fc7e0004eb0 con 0x7fc7fc06c380 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 >> v1:172.21.15.44:6790/0 conn(0x7fc7fc0b12d0 legacy=0x7fc7fc1e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 >> v1:172.21.15.44:6789/0 conn(0x7fc7fc0aedf0 legacy=0x7fc7fc1c0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 1 monclient: found mon.b 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc1c3270 con 0x7fc7fc06c380 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:42:41.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc7f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc80369a640 5 monclient: authenticate success, global_id 4381 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc80369a640 10 monclient: _renew_subs 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc80369a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7fc1c18e0 con 0x7fc7fc06c380 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.977+0000 7fc80369a640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.978+0000 7fc80369a640 10 monclient: _renew_subs 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.978+0000 7fc80369a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.978+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7fc1dad40 con 0x7fc7fc06c380 2024-03-25T17:42:41.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.978+0000 7fc80369a640 1 librados: init done 2024-03-25T17:42:41.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.979+0000 7fc80369a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:41.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.979+0000 7fc80369a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:41.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.979+0000 7fc80369a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:41.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:41.979+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7fc1c1ad0 con 0x7fc7fc06c380 2024-03-25T17:42:42.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.270+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7f80038a0 con 0x7fc7fc06c380 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.270+0000 7fc7f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.270+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc7f8003bb0 con 0x7fc7fc06c380 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.270+0000 7fc7f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.270+0000 7fc7f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.270+0000 7fc7f1ffb640 10 monclient: dump: 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:42.271 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:42.272 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:42.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.270+0000 7fc7f37fe640 4 set_mon_vals no callback set 2024-03-25T17:42:42.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.271+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fc7f8019b80 con 0x7fc7fc06c380 2024-03-25T17:42:42.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.272+0000 7fc800c35640 10 monclient: get_auth_request con 0x7fc7e006e170 auth_method 0 2024-03-25T17:42:42.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.272+0000 7fc7f1ffb640 10 client.4381.objecter ms_handle_connect 0x7fc7e006e170 2024-03-25T17:42:42.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.275+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fc7f80536b0 con 0x7fc7fc06c380 2024-03-25T17:42:42.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.275+0000 7fc7f1ffb640 10 client.4381.objecter ms_dispatch 0x7fc7fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:42:42.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.275+0000 7fc7f1ffb640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:42:42.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.275+0000 7fc7f1ffb640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:42:42.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.275+0000 7fc7f1ffb640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T17:42:42.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.276+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7f804f1c0 con 0x7fc7fc06c380 2024-03-25T17:42:42.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.276+0000 7fc7f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:42.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.276+0000 7fc7f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:42:42.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.491+0000 7fc80369a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:42.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.491+0000 7fc80369a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:42.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.491+0000 7fc80369a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:42.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.491+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7fc1c3650 con 0x7fc7fc06c380 2024-03-25T17:42:42.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.681+0000 7fc7f1ffb640 1 -- 172.21.15.44:0/223271441 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fc7f80510f0 con 0x7fc7fc06c380 2024-03-25T17:42:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.681+0000 7fc7f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.681+0000 7fc7f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:42:42.682 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:42:42.682 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 10 librados: watch_flush enter 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 10 librados: watch_flush exit 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 10 client.4381.objecter successfully canceled tick 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 >> v1:172.21.15.44:6800/3257043132 conn(0x7fc7e006e170 legacy=0x7fc7e0070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 10 monclient: shutdown 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 >> v1:172.21.15.111:6789/0 conn(0x7fc7fc06c380 legacy=0x7fc7fc1db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 shutdown_connections 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.683+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 >> 172.21.15.44:0/223271441 conn(0x7fc7fc0b5d10 msgr2=0x7fc7fc0af470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:42.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.684+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 shutdown_connections 2024-03-25T17:42:42.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.684+0000 7fc80369a640 1 -- 172.21.15.44:0/223271441 wait complete. 2024-03-25T17:42:42.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:42.684+0000 7fc80369a640 1 librados: shutdown 2024-03-25T17:42:42.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:42:49.503 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:42:49.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.619+0000 7fe5362c1640 10 monclient: get_monmap_and_config 2024-03-25T17:42:49.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.619+0000 7fe5362c1640 10 monclient: build_initial_monmap 2024-03-25T17:42:49.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.619+0000 7fe5362c1640 10 monclient: monmap: 2024-03-25T17:42:49.620 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:49.620 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:49.620 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:42:49.620724+0000 2024-03-25T17:42:49.620 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:42:49.620724+0000 2024-03-25T17:42:49.621 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:49.621 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:49.621 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:49.621 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:42:49.621 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:42:49.621 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.620+0000 7fe5362c1640 1 Processor -- start 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.620+0000 7fe5362c1640 1 -- start start 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.620+0000 7fe5362c1640 10 monclient: init 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 10 monclient(hunting): picked mon.noname-a con 0x7fe5300a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 10 monclient(hunting): picked mon.noname-b con 0x7fe53013f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 10 monclient(hunting): picked mon.noname-c con 0x7fe5301430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe530142520 con 0x7fe5300a98f0 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300b28a0 con 0x7fe5301430c0 2024-03-25T17:42:49.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300b2a30 con 0x7fe53013f710 2024-03-25T17:42:49.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:49.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.621+0000 7fe5362c1640 10 monclient(hunting): authenticate will time out at 6172.849463s 2024-03-25T17:42:49.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.623+0000 7fe52f7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fe53013f710 0x7fe5301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:45990/0 (socket says 172.21.15.44:45990) 2024-03-25T17:42:49.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.623+0000 7fe52f7fe640 1 -- 172.21.15.44:0/1677212577 learned_addr learned my addr 172.21.15.44:0/1677212577 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:49.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.636+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936157606 0 0) 0x7fe5300b2a30 con 0x7fe53013f710 2024-03-25T17:42:49.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.636+0000 7fe52effd640 10 monclient(hunting): _negotiate 2024-03-25T17:42:49.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.636+0000 7fe52effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:49.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.636+0000 7fe52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:49.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.636+0000 7fe52effd640 10 monclient(hunting): my global_id is 4382 2024-03-25T17:42:49.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.636+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe518003520 con 0x7fe53013f710 2024-03-25T17:42:49.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.646+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230164102 0 0) 0x7fe530142520 con 0x7fe5300a98f0 2024-03-25T17:42:49.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.646+0000 7fe52effd640 10 monclient(hunting): _negotiate 2024-03-25T17:42:49.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.646+0000 7fe52effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:49.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.646+0000 7fe52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:49.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.646+0000 7fe52effd640 10 monclient(hunting): my global_id is 4398 2024-03-25T17:42:49.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.646+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5300b2a30 con 0x7fe5300a98f0 2024-03-25T17:42:49.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.647+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228275274 0 0) 0x7fe5300b28a0 con 0x7fe5301430c0 2024-03-25T17:42:49.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.647+0000 7fe52effd640 10 monclient(hunting): _negotiate 2024-03-25T17:42:49.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.647+0000 7fe52effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:49.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.647+0000 7fe52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:49.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.647+0000 7fe52effd640 10 monclient(hunting): my global_id is 4384 2024-03-25T17:42:49.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.647+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe530142520 con 0x7fe5301430c0 2024-03-25T17:42:49.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4195130983 0 0) 0x7fe518003520 con 0x7fe53013f710 2024-03-25T17:42:49.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 >> v1:172.21.15.111:6789/0 conn(0x7fe5301430c0 legacy=0x7fe53014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:49.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 >> v1:172.21.15.44:6789/0 conn(0x7fe5300a98f0 legacy=0x7fe5300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:49.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 10 monclient: _finish_hunting 0 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 1 monclient: found mon.noname-b 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5300b44c0 con 0x7fe53013f710 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 10 monclient: _finish_auth 0 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 10 monclient: _check_auth_tickets 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.653+0000 7fe52effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.654+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe524001690 con 0x7fe53013f710 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.654+0000 7fe5362c1640 5 monclient: authenticate success, global_id 4382 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.654+0000 7fe5362c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.654+0000 7fe52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.654+0000 7fe52effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:42:49.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.654+0000 7fe52effd640 10 monclient: dump: 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:49.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.654+0000 7fe5362c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:49.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.672+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5240017f0 con 0x7fe53013f710 2024-03-25T17:42:49.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.672+0000 7fe52effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:49.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.672+0000 7fe52effd640 1 -- 172.21.15.44:0/1677212577 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe524001c00 con 0x7fe53013f710 2024-03-25T17:42:49.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.672+0000 7fe5362c1640 10 monclient: get_monmap_and_config success 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.672+0000 7fe5362c1640 4 set_mon_vals no callback set 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.672+0000 7fe52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.672+0000 7fe52effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.672+0000 7fe52effd640 10 monclient: dump: 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:49.673 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:49.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.673+0000 7fe5362c1640 10 monclient: shutdown 2024-03-25T17:42:49.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.673+0000 7fe5362c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:49.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.673+0000 7fe5362c1640 1 -- 172.21.15.44:0/1677212577 >> v1:172.21.15.44:6790/0 conn(0x7fe53013f710 legacy=0x7fe5301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:49.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.673+0000 7fe5362c1640 1 -- 172.21.15.44:0/1677212577 shutdown_connections 2024-03-25T17:42:49.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.673+0000 7fe5362c1640 1 -- 172.21.15.44:0/1677212577 >> 172.21.15.44:0/1677212577 conn(0x7fe5300b7ea0 msgr2=0x7fe5300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:49.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.673+0000 7fe5362c1640 1 -- 172.21.15.44:0/1677212577 shutdown_connections 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.673+0000 7fe5362c1640 1 -- 172.21.15.44:0/1677212577 wait complete. 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.673+0000 7fe5362c1640 10 monclient: build_initial_monmap 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.674+0000 7fe5362c1640 10 monclient: monmap: 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:42:49.676 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.674+0000 7fe5362c1640 1 librados: starting msgr at 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.674+0000 7fe5362c1640 1 librados: starting objecter 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.674+0000 7fe5362c1640 1 Processor -- start 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.674+0000 7fe5362c1640 1 -- start start 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.674+0000 7fe5362c1640 1 librados: setting wanted keys 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.674+0000 7fe5362c1640 1 librados: calling monclient init 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.674+0000 7fe5362c1640 10 monclient: init 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 10 monclient(hunting): picked mon.noname-b con 0x7fe5300a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 10 monclient(hunting): picked mon.noname-a con 0x7fe53013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 10 monclient(hunting): picked mon.noname-c con 0x7fe5301430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe530137df0 con 0x7fe53013f710 2024-03-25T17:42:49.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5301337a0 con 0x7fe5300a98f0 2024-03-25T17:42:49.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe530133930 con 0x7fe5301430c0 2024-03-25T17:42:49.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:49.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.675+0000 7fe5362c1640 10 monclient(hunting): authenticate will time out at 6172.903174s 2024-03-25T17:42:49.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.692+0000 7fe52f7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe53013f710 0x7fe5301376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52916/0 (socket says 172.21.15.44:52916) 2024-03-25T17:42:49.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.692+0000 7fe52f7fe640 1 -- 172.21.15.44:0/2870003734 learned_addr learned my addr 172.21.15.44:0/2870003734 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:49.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.693+0000 7fe52cff9640 10 client.?.objecter ms_handle_connect 0x7fe53013f710 2024-03-25T17:42:49.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.693+0000 7fe52cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:49.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.694+0000 7fe52cff9640 10 client.?.objecter ms_handle_connect 0x7fe5300a98f0 2024-03-25T17:42:49.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.694+0000 7fe52cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:49.696 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.696+0000 7fe52cff9640 10 client.?.objecter ms_handle_connect 0x7fe5301430c0 2024-03-25T17:42:49.696 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.696+0000 7fe52cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:49.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.701+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696853250 0 0) 0x7fe530133930 con 0x7fe5301430c0 2024-03-25T17:42:49.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.701+0000 7fe52cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:42:49.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.701+0000 7fe52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:49.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.701+0000 7fe52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:49.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.701+0000 7fe52cff9640 10 monclient(hunting): my global_id is 4385 2024-03-25T17:42:49.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.701+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c004eb0 con 0x7fe5301430c0 2024-03-25T17:42:49.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.702+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950566643 0 0) 0x7fe5301337a0 con 0x7fe5300a98f0 2024-03-25T17:42:49.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.702+0000 7fe52cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:42:49.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.702+0000 7fe52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:49.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.702+0000 7fe52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:49.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.702+0000 7fe52cff9640 10 monclient(hunting): my global_id is 4387 2024-03-25T17:42:49.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.702+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe530133930 con 0x7fe5300a98f0 2024-03-25T17:42:49.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.704+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122418275 0 0) 0x7fe530137df0 con 0x7fe53013f710 2024-03-25T17:42:49.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.704+0000 7fe52cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:42:49.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.704+0000 7fe52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:49.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.704+0000 7fe52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:49.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.704+0000 7fe52cff9640 10 monclient(hunting): my global_id is 4401 2024-03-25T17:42:49.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.704+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5301337a0 con 0x7fe53013f710 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3127134649 0 0) 0x7fe51c004eb0 con 0x7fe5301430c0 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe530137df0 con 0x7fe5301430c0 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe528001bd0 con 0x7fe5301430c0 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 10 monclient(hunting): dump: 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:49.719 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 201815890 0 0) 0x7fe530133930 con 0x7fe5300a98f0 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe51c004eb0 con 0x7fe5300a98f0 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe5200018f0 con 0x7fe5300a98f0 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.718+0000 7fe52cff9640 10 monclient(hunting): dump: 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:49.720 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.721 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.721 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:49.721 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:49.721 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:49.721 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:49.721 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:49.721 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:49.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.722+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3319121294 0 0) 0x7fe5301337a0 con 0x7fe53013f710 2024-03-25T17:42:49.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.722+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe530133930 con 0x7fe53013f710 2024-03-25T17:42:49.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.722+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe524001b00 con 0x7fe53013f710 2024-03-25T17:42:49.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.722+0000 7fe52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:49.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.723+0000 7fe52cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:42:49.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.723+0000 7fe52cff9640 10 monclient(hunting): dump: 2024-03-25T17:42:49.723 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:49.723 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:49.724 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.724 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.724 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:49.724 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:49.724 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:49.724 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:49.724 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:49.724 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4028466395 0 0) 0x7fe51c004eb0 con 0x7fe5300a98f0 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 >> v1:172.21.15.44:6790/0 conn(0x7fe5301430c0 legacy=0x7fe530133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 >> v1:172.21.15.44:6789/0 conn(0x7fe53013f710 legacy=0x7fe5301376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 1 monclient: found mon.b 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5301367d0 con 0x7fe5300a98f0 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 10 monclient: _finish_auth 0 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe52cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 5 monclient: authenticate success, global_id 4387 2024-03-25T17:42:49.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 10 monclient: _renew_subs 2024-03-25T17:42:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5301369e0 con 0x7fe5300a98f0 2024-03-25T17:42:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:42:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 10 monclient: _renew_subs 2024-03-25T17:42:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53013b640 con 0x7fe5300a98f0 2024-03-25T17:42:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.728+0000 7fe5362c1640 1 librados: init done 2024-03-25T17:42:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.729+0000 7fe5362c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:49.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.729+0000 7fe5362c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:49.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.729+0000 7fe5362c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:49.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.729+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe530074cc0 con 0x7fe5300a98f0 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.747+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe520001d60 con 0x7fe5300a98f0 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.747+0000 7fe52cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.747+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe520003790 con 0x7fe5300a98f0 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.747+0000 7fe52cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.747+0000 7fe52cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.747+0000 7fe52cff9640 10 monclient: dump: 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:49.748 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:49.749 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:49.749 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:49.749 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:49.749 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:49.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.747+0000 7fe52e7fc640 4 set_mon_vals no callback set 2024-03-25T17:42:49.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.748+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fe5200197e0 con 0x7fe5300a98f0 2024-03-25T17:42:49.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.749+0000 7fe52f7fe640 10 monclient: get_auth_request con 0x7fe51c06e110 auth_method 0 2024-03-25T17:42:49.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.749+0000 7fe52cff9640 10 client.4387.objecter ms_handle_connect 0x7fe51c06e110 2024-03-25T17:42:49.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.751+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fe520053500 con 0x7fe5300a98f0 2024-03-25T17:42:49.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.751+0000 7fe52cff9640 10 client.4387.objecter ms_dispatch 0x7fe530000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:42:49.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.751+0000 7fe52cff9640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:42:49.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.751+0000 7fe52cff9640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:42:49.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:49.751+0000 7fe52cff9640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-25T17:42:50.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.015+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe52004ee40 con 0x7fe5300a98f0 2024-03-25T17:42:50.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.015+0000 7fe52cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:50.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.015+0000 7fe52cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:42:50.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.264+0000 7fe5362c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:50.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.264+0000 7fe5362c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:50.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.264+0000 7fe5362c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:42:50.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.264+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5300627b0 con 0x7fe5300a98f0 2024-03-25T17:42:50.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.351+0000 7fe52cff9640 1 -- 172.21.15.44:0/2870003734 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fe520050f40 con 0x7fe5300a98f0 2024-03-25T17:42:50.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.351+0000 7fe52cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:50.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.351+0000 7fe52cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:42:50.352 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:42:50.352 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 10 librados: watch_flush enter 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 10 librados: watch_flush exit 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 10 client.4387.objecter successfully canceled tick 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 >> v1:172.21.15.44:6800/3257043132 conn(0x7fe51c06e110 legacy=0x7fe51c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 10 monclient: shutdown 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 >> v1:172.21.15.111:6789/0 conn(0x7fe5300a98f0 legacy=0x7fe5301392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 shutdown_connections 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 >> 172.21.15.44:0/2870003734 conn(0x7fe5300b7ea0 msgr2=0x7fe530074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:50.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 shutdown_connections 2024-03-25T17:42:50.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 1 -- 172.21.15.44:0/2870003734 wait complete. 2024-03-25T17:42:50.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:50.354+0000 7fe5362c1640 1 librados: shutdown 2024-03-25T17:42:50.366 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:42:57.173 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.276+0000 7f61a68d6640 10 monclient: get_monmap_and_config 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.276+0000 7f61a68d6640 10 monclient: build_initial_monmap 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.276+0000 7f61a68d6640 10 monclient: monmap: 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:42:57.277566+0000 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:42:57.277566+0000 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:57.277 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:57.278 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:42:57.278 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:42:57.278 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:57.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.277+0000 7f61a68d6640 1 Processor -- start 2024-03-25T17:42:57.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.277+0000 7f61a68d6640 1 -- start start 2024-03-25T17:42:57.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.277+0000 7f61a68d6640 10 monclient: init 2024-03-25T17:42:57.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:57.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:42:57.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 10 monclient(hunting): picked mon.noname-c con 0x7f61a00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:57.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 10 monclient(hunting): picked mon.noname-a con 0x7f61a013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:57.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 10 monclient(hunting): picked mon.noname-b con 0x7f61a01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:57.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a0142520 con 0x7f61a013f710 2024-03-25T17:42:57.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a00b28a0 con 0x7f61a00a98f0 2024-03-25T17:42:57.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a00b2a30 con 0x7f61a01430c0 2024-03-25T17:42:57.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:57.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.278+0000 7f61a68d6640 10 monclient(hunting): authenticate will time out at 6180.506208s 2024-03-25T17:42:57.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.287+0000 7f619ffff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f61a00a98f0 0x7f61a00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60038/0 (socket says 172.21.15.44:60038) 2024-03-25T17:42:57.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.287+0000 7f619ffff640 1 -- 172.21.15.44:0/861748086 learned_addr learned my addr 172.21.15.44:0/861748086 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:57.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.302+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174447865 0 0) 0x7f61a00b28a0 con 0x7f61a00a98f0 2024-03-25T17:42:57.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.302+0000 7f619effd640 10 monclient(hunting): _negotiate 2024-03-25T17:42:57.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.302+0000 7f619effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:57.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.302+0000 7f619effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:57.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.302+0000 7f619effd640 10 monclient(hunting): my global_id is 4390 2024-03-25T17:42:57.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.302+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188003520 con 0x7f61a00a98f0 2024-03-25T17:42:57.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.307+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381071136 0 0) 0x7f61a0142520 con 0x7f61a013f710 2024-03-25T17:42:57.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.307+0000 7f619effd640 10 monclient(hunting): _negotiate 2024-03-25T17:42:57.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.307+0000 7f619effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:57.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.307+0000 7f619effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:57.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.307+0000 7f619effd640 10 monclient(hunting): my global_id is 4404 2024-03-25T17:42:57.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.307+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a00b28a0 con 0x7f61a013f710 2024-03-25T17:42:57.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.325+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554368239 0 0) 0x7f61a00b2a30 con 0x7f61a01430c0 2024-03-25T17:42:57.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.325+0000 7f619effd640 10 monclient(hunting): _negotiate 2024-03-25T17:42:57.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.325+0000 7f619effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:57.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.325+0000 7f619effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:57.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.325+0000 7f619effd640 10 monclient(hunting): my global_id is 4388 2024-03-25T17:42:57.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.325+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0142520 con 0x7f61a01430c0 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43236294 0 0) 0x7f61a00b28a0 con 0x7f61a013f710 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 >> v1:172.21.15.44:6790/0 conn(0x7f61a01430c0 legacy=0x7f61a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 >> v1:172.21.15.111:6789/0 conn(0x7f61a00a98f0 legacy=0x7f61a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 10 monclient: _finish_hunting 0 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 1 monclient: found mon.noname-a 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a00b44c0 con 0x7f61a013f710 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 10 monclient: _finish_auth 0 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 10 monclient: _check_auth_tickets 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6194001670 con 0x7f61a013f710 2024-03-25T17:42:57.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f619effd640 10 monclient: dump: 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f61a68d6640 5 monclient: authenticate success, global_id 4404 2024-03-25T17:42:57.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.326+0000 7f61a68d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:42:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.335+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61940017d0 con 0x7f61a013f710 2024-03-25T17:42:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.335+0000 7f619effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.335+0000 7f619effd640 1 -- 172.21.15.44:0/861748086 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6194001be0 con 0x7f61a013f710 2024-03-25T17:42:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.335+0000 7f61a68d6640 10 monclient: get_monmap_and_config success 2024-03-25T17:42:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.335+0000 7f61a68d6640 4 set_mon_vals no callback set 2024-03-25T17:42:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.335+0000 7f619effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.335+0000 7f619effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:42:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.335+0000 7f619effd640 10 monclient: dump: 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:57.336 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.336+0000 7f61a68d6640 10 monclient: shutdown 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.336+0000 7f61a68d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.336+0000 7f61a68d6640 1 -- 172.21.15.44:0/861748086 >> v1:172.21.15.44:6789/0 conn(0x7f61a013f710 legacy=0x7f61a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.336+0000 7f61a68d6640 1 -- 172.21.15.44:0/861748086 shutdown_connections 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.336+0000 7f61a68d6640 1 -- 172.21.15.44:0/861748086 >> 172.21.15.44:0/861748086 conn(0x7f61a00b7ea0 msgr2=0x7f61a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.336+0000 7f61a68d6640 1 -- 172.21.15.44:0/861748086 shutdown_connections 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.336+0000 7f61a68d6640 1 -- 172.21.15.44:0/861748086 wait complete. 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.337+0000 7f61a68d6640 10 monclient: build_initial_monmap 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.337+0000 7f61a68d6640 10 monclient: monmap: 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:42:57.338 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.337+0000 7f61a68d6640 1 librados: starting msgr at 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.337+0000 7f61a68d6640 1 librados: starting objecter 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.337+0000 7f61a68d6640 1 Processor -- start 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.338+0000 7f61a68d6640 1 -- start start 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.338+0000 7f61a68d6640 1 librados: setting wanted keys 2024-03-25T17:42:57.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.338+0000 7f61a68d6640 1 librados: calling monclient init 2024-03-25T17:42:57.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.338+0000 7f61a68d6640 10 monclient: init 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.338+0000 7f61a68d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.338+0000 7f61a68d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.338+0000 7f61a68d6640 10 monclient(hunting): picked mon.noname-b con 0x7f61a00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.339+0000 7f61a68d6640 10 monclient(hunting): picked mon.noname-a con 0x7f61a013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.339+0000 7f61a68d6640 10 monclient(hunting): picked mon.noname-c con 0x7f61a01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.339+0000 7f61a68d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a0136750 con 0x7f61a013f710 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.339+0000 7f61a68d6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a01dac30 con 0x7f61a00a98f0 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.339+0000 7f61a68d6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a01399c0 con 0x7f61a01430c0 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.339+0000 7f61a68d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:42:57.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.339+0000 7f61a68d6640 10 monclient(hunting): authenticate will time out at 6180.566908s 2024-03-25T17:42:57.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.346+0000 7f61a4e73640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f61a01430c0 0x7f61a01d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:50812/0 (socket says 172.21.15.44:50812) 2024-03-25T17:42:57.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.346+0000 7f61a4e73640 1 -- 172.21.15.44:0/800013471 learned_addr learned my addr 172.21.15.44:0/800013471 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:42:57.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.348+0000 7f619cff9640 10 client.?.objecter ms_handle_connect 0x7f61a01430c0 2024-03-25T17:42:57.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.348+0000 7f619cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:57.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.354+0000 7f619cff9640 10 client.?.objecter ms_handle_connect 0x7f61a013f710 2024-03-25T17:42:57.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.354+0000 7f619cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:57.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.354+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716810108 0 0) 0x7f61a01399c0 con 0x7f61a01430c0 2024-03-25T17:42:57.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.354+0000 7f619cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:42:57.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.354+0000 7f619cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:57.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.354+0000 7f619cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:57.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.354+0000 7f619cff9640 10 monclient(hunting): my global_id is 4391 2024-03-25T17:42:57.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.354+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618c004eb0 con 0x7f61a01430c0 2024-03-25T17:42:57.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.362+0000 7f619cff9640 10 client.?.objecter ms_handle_connect 0x7f61a00a98f0 2024-03-25T17:42:57.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.362+0000 7f619cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.366+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2102657606 0 0) 0x7f618c004eb0 con 0x7f61a01430c0 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.366+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61a01399c0 con 0x7f61a01430c0 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.366+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6198001c70 con 0x7f61a01430c0 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.366+0000 7f619cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.366+0000 7f619cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.366+0000 7f619cff9640 10 monclient(hunting): dump: 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:57.367 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:57.368 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.366+0000 7f619cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.368+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828558944 0 0) 0x7f61a0136750 con 0x7f61a013f710 2024-03-25T17:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.368+0000 7f619cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.368+0000 7f619cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.368+0000 7f619cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.368+0000 7f619cff9640 10 monclient(hunting): my global_id is 4407 2024-03-25T17:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.368+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618c004eb0 con 0x7f61a013f710 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3420734052 0 0) 0x7f61a01399c0 con 0x7f61a01430c0 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 >> v1:172.21.15.111:6789/0 conn(0x7f61a00a98f0 legacy=0x7f61a01386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 >> v1:172.21.15.44:6789/0 conn(0x7f61a013f710 legacy=0x7f61a01db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 1 monclient: found mon.c 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a01db860 con 0x7f61a01430c0 2024-03-25T17:42:57.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 10 monclient: _finish_auth 0 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f619cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 5 monclient: authenticate success, global_id 4391 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 10 monclient: _renew_subs 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a01d41e0 con 0x7f61a01430c0 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 10 monclient: _renew_subs 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a01d5150 con 0x7f61a01430c0 2024-03-25T17:42:57.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.379+0000 7f61a68d6640 1 librados: init done 2024-03-25T17:42:57.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.380+0000 7f61a68d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:57.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.380+0000 7f61a68d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:57.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.380+0000 7f61a68d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:57.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.380+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a01daa90 con 0x7f61a01430c0 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.704+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6198003990 con 0x7f61a01430c0 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.704+0000 7f619cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.704+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6198003c40 con 0x7f61a01430c0 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.704+0000 7f619cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.704+0000 7f619cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.704+0000 7f619cff9640 10 monclient: dump: 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:42:57.705 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:57.706 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:42:57.706 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:42:57.706 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:42:57.706 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:42:57.706 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:42:57.706 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:42:57.706 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:42:57.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.704+0000 7f619e7fc640 4 set_mon_vals no callback set 2024-03-25T17:42:57.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.705+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f6198003e60 con 0x7f61a01430c0 2024-03-25T17:42:57.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.706+0000 7f619ffff640 10 monclient: get_auth_request con 0x7f618c06dd90 auth_method 0 2024-03-25T17:42:57.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.706+0000 7f619cff9640 10 client.4391.objecter ms_handle_connect 0x7f618c06dd90 2024-03-25T17:42:57.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.713+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f6198054910 con 0x7f61a01430c0 2024-03-25T17:42:57.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.713+0000 7f619cff9640 10 client.4391.objecter ms_dispatch 0x7f61a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:42:57.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.713+0000 7f619cff9640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:42:57.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.713+0000 7f619cff9640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:42:57.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.713+0000 7f619cff9640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-25T17:42:57.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.713+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6198050420 con 0x7f61a01430c0 2024-03-25T17:42:57.715 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.713+0000 7f619cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:42:57.715 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.713+0000 7f619cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:42:57.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.946+0000 7f61a68d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:57.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.946+0000 7f61a68d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:57.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.946+0000 7f61a68d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:42:57.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.946+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61a01db9f0 con 0x7f61a01430c0 2024-03-25T17:42:57.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.975+0000 7f619cff9640 1 -- 172.21.15.44:0/800013471 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f61a01db9f0 con 0x7f61a01430c0 2024-03-25T17:42:57.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.975+0000 7f619cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:42:57.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.975+0000 7f619cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:42:57.976 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:42:57.976 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:42:57.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.977+0000 7f61a68d6640 10 librados: watch_flush enter 2024-03-25T17:42:57.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.977+0000 7f61a68d6640 10 librados: watch_flush exit 2024-03-25T17:42:57.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.977+0000 7f61a68d6640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-25T17:42:57.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.977+0000 7f61a68d6640 10 client.4391.objecter successfully canceled tick 2024-03-25T17:42:57.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 >> v1:172.21.15.44:6800/3257043132 conn(0x7f618c06dd90 legacy=0x7f618c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:57.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 10 monclient: shutdown 2024-03-25T17:42:57.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:42:57.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 >> v1:172.21.15.44:6790/0 conn(0x7f61a01430c0 legacy=0x7f61a01d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:42:57.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 shutdown_connections 2024-03-25T17:42:57.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 >> 172.21.15.44:0/800013471 conn(0x7f61a00b7ea0 msgr2=0x7f61a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:42:57.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 shutdown_connections 2024-03-25T17:42:57.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 1 -- 172.21.15.44:0/800013471 wait complete. 2024-03-25T17:42:57.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:42:57.978+0000 7f61a68d6640 1 librados: shutdown 2024-03-25T17:42:57.991 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:43:04.797 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:43:04.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.910+0000 7fbfbc4c1640 10 monclient: get_monmap_and_config 2024-03-25T17:43:04.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.910+0000 7fbfbc4c1640 10 monclient: build_initial_monmap 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.910+0000 7fbfbc4c1640 10 monclient: monmap: 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:43:04.912168+0000 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:43:04.912168+0000 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:43:04.912 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:04.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.910+0000 7fbfbc4c1640 1 Processor -- start 2024-03-25T17:43:04.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.910+0000 7fbfbc4c1640 1 -- start start 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.910+0000 7fbfbc4c1640 10 monclient: init 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 10 monclient(hunting): picked mon.noname-c con 0x7fbfb40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb40b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb414be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40af940 con 0x7fbfb40b1730 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40b0a70 con 0x7fbfb40b4d60 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40b0d50 con 0x7fbfb414be90 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:04.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.911+0000 7fbfbc4c1640 10 monclient(hunting): authenticate will time out at 6188.140573s 2024-03-25T17:43:04.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.920+0000 7fbfbaa5e640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fbfb414be90 0x7fbfb414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:41942/0 (socket says 172.21.15.44:41942) 2024-03-25T17:43:04.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.920+0000 7fbfbaa5e640 1 -- 172.21.15.44:0/516052236 learned_addr learned my addr 172.21.15.44:0/516052236 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:04.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055109508 0 0) 0x7fbfb40b0d50 con 0x7fbfb414be90 2024-03-25T17:43:04.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb925b640 10 monclient(hunting): _negotiate 2024-03-25T17:43:04.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb925b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:04.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb925b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:04.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb925b640 10 monclient(hunting): my global_id is 4394 2024-03-25T17:43:04.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa0003520 con 0x7fbfb414be90 2024-03-25T17:43:04.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb9a5c640 1 -- 172.21.15.44:0/516052236 >> v1:172.21.15.44:6789/0 conn(0x7fbfb40b1730 legacy=0x7fbfb40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T17:43:04.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb9a5c640 1 -- 172.21.15.44:0/516052236 >> v1:172.21.15.44:6789/0 conn(0x7fbfb40b1730 legacy=0x7fbfb40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:43:04.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb9a5c640 1 --1- 172.21.15.44:0/516052236 >> v1:172.21.15.44:6789/0 conn(0x7fbfb40b1730 0x7fbfb40b1be0 :-1 s=OPENED pgs=26944 cs=1 l=1).handle_message read tag failed 2024-03-25T17:43:04.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb9a5c640 1 --1- 172.21.15.44:0/516052236 >> v1:172.21.15.44:6789/0 conn(0x7fbfb40b1730 0x7fbfb40b1be0 :-1 s=OPENED pgs=26944 cs=1 l=1).fault on lossy channel, failing 2024-03-25T17:43:04.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.925+0000 7fbfb925b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.44:6789/0 2024-03-25T17:43:04.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.932+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145937116 0 0) 0x7fbfb40b0a70 con 0x7fbfb40b4d60 2024-03-25T17:43:04.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.932+0000 7fbfb925b640 10 monclient(hunting): _negotiate 2024-03-25T17:43:04.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.932+0000 7fbfb925b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:04.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.932+0000 7fbfb925b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:04.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.932+0000 7fbfb925b640 10 monclient(hunting): my global_id is 4396 2024-03-25T17:43:04.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.932+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb40b0d50 con 0x7fbfb40b4d60 2024-03-25T17:43:04.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3828527313 0 0) 0x7fbfb40b0d50 con 0x7fbfb40b4d60 2024-03-25T17:43:04.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 >> v1:172.21.15.44:6790/0 conn(0x7fbfb414be90 legacy=0x7fbfb414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:04.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 >> v1:172.21.15.44:6789/0 conn(0x7fbfb40b1730 legacy=0x7fbfb40b1be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 10 monclient: _finish_hunting 0 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 1 monclient: found mon.noname-c 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb40b1030 con 0x7fbfb40b4d60 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 10 monclient: _finish_auth 0 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 10 monclient: _check_auth_tickets 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbfb0001560 con 0x7fbfb40b4d60 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfb925b640 10 monclient: dump: 2024-03-25T17:43:04.954 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfbc4c1640 5 monclient: authenticate success, global_id 4396 2024-03-25T17:43:04.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.952+0000 7fbfbc4c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:04.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.958+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfb00016c0 con 0x7fbfb40b4d60 2024-03-25T17:43:04.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.958+0000 7fbfb925b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:04.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.958+0000 7fbfb925b640 1 -- 172.21.15.44:0/516052236 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbfb0001ad0 con 0x7fbfb40b4d60 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.958+0000 7fbfbc4c1640 10 monclient: get_monmap_and_config success 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.958+0000 7fbfbc4c1640 4 set_mon_vals no callback set 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.958+0000 7fbfb925b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.958+0000 7fbfb925b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.958+0000 7fbfb925b640 10 monclient: dump: 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:04.960 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:04.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 10 monclient: shutdown 2024-03-25T17:43:04.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:04.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/516052236 >> v1:172.21.15.111:6789/0 conn(0x7fbfb40b4d60 legacy=0x7fbfb413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:04.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/516052236 shutdown_connections 2024-03-25T17:43:04.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/516052236 >> 172.21.15.44:0/516052236 conn(0x7fbfb40617a0 msgr2=0x7fbfb4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:04.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/516052236 shutdown_connections 2024-03-25T17:43:04.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/516052236 wait complete. 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 10 monclient: build_initial_monmap 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.959+0000 7fbfbc4c1640 10 monclient: monmap: 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 librados: starting msgr at 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 librados: starting objecter 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 Processor -- start 2024-03-25T17:43:04.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 -- start start 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 librados: setting wanted keys 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 librados: calling monclient init 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 10 monclient: init 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb40b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 10 monclient(hunting): picked mon.noname-c con 0x7fbfb414be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb413a8c0 con 0x7fbfb40b1730 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb41d8640 con 0x7fbfb40b4d60 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb41d7730 con 0x7fbfb414be90 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:04.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.960+0000 7fbfbc4c1640 10 monclient(hunting): authenticate will time out at 6188.189642s 2024-03-25T17:43:04.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.974+0000 7fbfba25d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbfb40b1730 0x7fbfb41d7f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:39662/0 (socket says 172.21.15.44:39662) 2024-03-25T17:43:04.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.974+0000 7fbfba25d640 1 -- 172.21.15.44:0/1638551649 learned_addr learned my addr 172.21.15.44:0/1638551649 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:04.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.976+0000 7fbfaaffd640 10 client.?.objecter ms_handle_connect 0x7fbfb40b4d60 2024-03-25T17:43:04.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.976+0000 7fbfaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:04.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.977+0000 7fbfaaffd640 10 client.?.objecter ms_handle_connect 0x7fbfb40b1730 2024-03-25T17:43:04.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.977+0000 7fbfaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:04.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.983+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772695345 0 0) 0x7fbfb413a8c0 con 0x7fbfb40b1730 2024-03-25T17:43:04.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.983+0000 7fbfaaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:04.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.983+0000 7fbfaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:04.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.983+0000 7fbfaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:04.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.983+0000 7fbfaaffd640 10 monclient(hunting): my global_id is 4410 2024-03-25T17:43:04.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.983+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf98004eb0 con 0x7fbfb40b1730 2024-03-25T17:43:04.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.984+0000 7fbfaaffd640 10 client.?.objecter ms_handle_connect 0x7fbfb414be90 2024-03-25T17:43:04.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.984+0000 7fbfaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:04.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.992+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860004267 0 0) 0x7fbfb41d8640 con 0x7fbfb40b4d60 2024-03-25T17:43:04.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.992+0000 7fbfaaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:04.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.992+0000 7fbfaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:04.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.992+0000 7fbfaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:04.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.992+0000 7fbfaaffd640 10 monclient(hunting): my global_id is 4399 2024-03-25T17:43:04.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.992+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb413a8c0 con 0x7fbfb40b4d60 2024-03-25T17:43:04.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.993+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3096809867 0 0) 0x7fbf98004eb0 con 0x7fbfb40b1730 2024-03-25T17:43:04.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.994+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfb41d8640 con 0x7fbfb40b1730 2024-03-25T17:43:04.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.994+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbfb0001ad0 con 0x7fbfb40b1730 2024-03-25T17:43:04.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.994+0000 7fbfaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:04.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.994+0000 7fbfaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:43:04.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.994+0000 7fbfaaffd640 10 monclient(hunting): dump: 2024-03-25T17:43:04.995 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:04.995 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:04.994+0000 7fbfaaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3565535098 0 0) 0x7fbfb41d8640 con 0x7fbfb40b1730 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 >> v1:172.21.15.44:6790/0 conn(0x7fbfb414be90 legacy=0x7fbfb41e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 >> v1:172.21.15.111:6789/0 conn(0x7fbfb40b4d60 legacy=0x7fbfb41415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 10 monclient: _finish_hunting 0 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 1 monclient: found mon.a 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb41d7a10 con 0x7fbfb40b1730 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 10 monclient: _finish_auth 0 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 10 monclient: _check_auth_tickets 2024-03-25T17:43:05.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 5 monclient: authenticate success, global_id 4410 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 10 monclient: _renew_subs 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb41427c0 con 0x7fbfb40b1730 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 10 monclient: _renew_subs 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb4143270 con 0x7fbfb40b1730 2024-03-25T17:43:05.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.003+0000 7fbfbc4c1640 1 librados: init done 2024-03-25T17:43:05.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.004+0000 7fbfbc4c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:05.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.004+0000 7fbfbc4c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:05.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.004+0000 7fbfbc4c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:05.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.004+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb40627b0 con 0x7fbfb40b1730 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.326+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfb00038a0 con 0x7fbfb40b1730 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.326+0000 7fbfaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.326+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbfb0003c70 con 0x7fbfb40b1730 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.326+0000 7fbfaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.326+0000 7fbfaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.326+0000 7fbfaaffd640 10 monclient: dump: 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:05.328 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:05.329 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:05.329 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:05.329 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:05.329 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:05.329 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:05.329 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:05.329 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:05.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.326+0000 7fbfb8a5a640 4 set_mon_vals no callback set 2024-03-25T17:43:05.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.327+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fbfb0003e90 con 0x7fbfb40b1730 2024-03-25T17:43:05.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.328+0000 7fbfb9a5c640 10 monclient: get_auth_request con 0x7fbf9806dda0 auth_method 0 2024-03-25T17:43:05.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.328+0000 7fbfaaffd640 10 client.4410.objecter ms_handle_connect 0x7fbf9806dda0 2024-03-25T17:43:05.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.334+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fbfb00548c0 con 0x7fbfb40b1730 2024-03-25T17:43:05.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.334+0000 7fbfaaffd640 10 client.4410.objecter ms_dispatch 0x7fbfb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:43:05.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.334+0000 7fbfaaffd640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:43:05.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.334+0000 7fbfaaffd640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:43:05.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.334+0000 7fbfaaffd640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T17:43:05.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.334+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfb00503d0 con 0x7fbfb40b1730 2024-03-25T17:43:05.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.334+0000 7fbfaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:05.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.334+0000 7fbfaaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:43:05.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.549+0000 7fbfbc4c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:05.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.549+0000 7fbfbc4c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:05.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.549+0000 7fbfbc4c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:05.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.549+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfb4069f40 con 0x7fbfb40b1730 2024-03-25T17:43:05.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.578+0000 7fbfaaffd640 1 -- 172.21.15.44:0/1638551649 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fbfb0052300 con 0x7fbfb40b1730 2024-03-25T17:43:05.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.578+0000 7fbfaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:05.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.578+0000 7fbfaaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:43:05.579 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:43:05.580 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:43:05.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 10 librados: watch_flush enter 2024-03-25T17:43:05.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 10 librados: watch_flush exit 2024-03-25T17:43:05.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T17:43:05.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 10 client.4410.objecter successfully canceled tick 2024-03-25T17:43:05.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 >> v1:172.21.15.44:6800/3257043132 conn(0x7fbf9806dda0 legacy=0x7fbf98070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:05.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 10 monclient: shutdown 2024-03-25T17:43:05.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:05.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 >> v1:172.21.15.44:6789/0 conn(0x7fbfb40b1730 legacy=0x7fbfb41d7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:05.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 shutdown_connections 2024-03-25T17:43:05.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 >> 172.21.15.44:0/1638551649 conn(0x7fbfb40617a0 msgr2=0x7fbfb40b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:05.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 shutdown_connections 2024-03-25T17:43:05.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 1 -- 172.21.15.44:0/1638551649 wait complete. 2024-03-25T17:43:05.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:05.580+0000 7fbfbc4c1640 1 librados: shutdown 2024-03-25T17:43:05.593 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:43:12.401 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:43:12.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.523+0000 7f3f5e681640 10 monclient: get_monmap_and_config 2024-03-25T17:43:12.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.523+0000 7f3f5e681640 10 monclient: build_initial_monmap 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.523+0000 7f3f5e681640 10 monclient: monmap: 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:43:12.524286+0000 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:43:12.524286+0000 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:43:12.524 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:12.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 1 Processor -- start 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 1 -- start start 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 10 monclient: init 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 10 monclient(hunting): picked mon.noname-c con 0x7f3f580a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 10 monclient(hunting): picked mon.noname-b con 0x7f3f5813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 10 monclient(hunting): picked mon.noname-a con 0x7f3f581430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f58142520 con 0x7f3f581430c0 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f580b28a0 con 0x7f3f580a98f0 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f580b2a30 con 0x7f3f5813f710 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:12.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.524+0000 7f3f5e681640 10 monclient(hunting): authenticate will time out at 6195.752653s 2024-03-25T17:43:12.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.531+0000 7f3f577fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f3f5813f710 0x7f3f581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:42042/0 (socket says 172.21.15.44:42042) 2024-03-25T17:43:12.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.531+0000 7f3f577fe640 1 -- 172.21.15.44:0/3787459149 learned_addr learned my addr 172.21.15.44:0/3787459149 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:12.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.546+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388841354 0 0) 0x7f3f580b2a30 con 0x7f3f5813f710 2024-03-25T17:43:12.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.546+0000 7f3f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:12.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.546+0000 7f3f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:12.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.546+0000 7f3f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:12.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.546+0000 7f3f56ffd640 10 monclient(hunting): my global_id is 4400 2024-03-25T17:43:12.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.546+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f40003520 con 0x7f3f5813f710 2024-03-25T17:43:12.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.554+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186638503 0 0) 0x7f3f580b28a0 con 0x7f3f580a98f0 2024-03-25T17:43:12.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.554+0000 7f3f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:12.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.554+0000 7f3f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:12.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.554+0000 7f3f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:12.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.554+0000 7f3f56ffd640 10 monclient(hunting): my global_id is 4402 2024-03-25T17:43:12.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.554+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f580b2a30 con 0x7f3f580a98f0 2024-03-25T17:43:12.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977523854 0 0) 0x7f3f580b2a30 con 0x7f3f580a98f0 2024-03-25T17:43:12.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 >> v1:172.21.15.44:6790/0 conn(0x7f3f5813f710 legacy=0x7f3f581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:12.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 >> v1:172.21.15.44:6789/0 conn(0x7f3f581430c0 legacy=0x7f3f5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:12.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:43:12.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 1 monclient: found mon.noname-c 2024-03-25T17:43:12.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:12.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f580b44c0 con 0x7f3f580a98f0 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 10 monclient: _finish_auth 0 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f3f48001670 con 0x7f3f580a98f0 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f56ffd640 10 monclient: dump: 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:12.562 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:12.563 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:12.563 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:12.563 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:12.563 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:12.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f5e681640 5 monclient: authenticate success, global_id 4402 2024-03-25T17:43:12.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.560+0000 7f3f5e681640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:12.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.575+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f480017d0 con 0x7f3f580a98f0 2024-03-25T17:43:12.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.575+0000 7f3f56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.575+0000 7f3f56ffd640 1 -- 172.21.15.44:0/3787459149 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f3f48001b60 con 0x7f3f580a98f0 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.575+0000 7f3f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.575+0000 7f3f56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.575+0000 7f3f56ffd640 10 monclient: dump: 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.575+0000 7f3f5e681640 10 monclient: get_monmap_and_config success 2024-03-25T17:43:12.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.575+0000 7f3f5e681640 4 set_mon_vals no callback set 2024-03-25T17:43:12.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.576+0000 7f3f5e681640 10 monclient: shutdown 2024-03-25T17:43:12.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.576+0000 7f3f5e681640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:12.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.576+0000 7f3f5e681640 1 -- 172.21.15.44:0/3787459149 >> v1:172.21.15.111:6789/0 conn(0x7f3f580a98f0 legacy=0x7f3f580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:12.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.576+0000 7f3f5e681640 1 -- 172.21.15.44:0/3787459149 shutdown_connections 2024-03-25T17:43:12.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.576+0000 7f3f5e681640 1 -- 172.21.15.44:0/3787459149 >> 172.21.15.44:0/3787459149 conn(0x7f3f580b7ea0 msgr2=0x7f3f580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:12.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.576+0000 7f3f5e681640 1 -- 172.21.15.44:0/3787459149 shutdown_connections 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.576+0000 7f3f5e681640 1 -- 172.21.15.44:0/3787459149 wait complete. 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.577+0000 7f3f5e681640 10 monclient: build_initial_monmap 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.577+0000 7f3f5e681640 10 monclient: monmap: 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.577+0000 7f3f5e681640 1 librados: starting msgr at 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.577+0000 7f3f5e681640 1 librados: starting objecter 2024-03-25T17:43:12.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.577+0000 7f3f5e681640 1 Processor -- start 2024-03-25T17:43:12.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 1 -- start start 2024-03-25T17:43:12.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 1 librados: setting wanted keys 2024-03-25T17:43:12.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 1 librados: calling monclient init 2024-03-25T17:43:12.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 10 monclient: init 2024-03-25T17:43:12.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:12.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:43:12.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 10 monclient(hunting): picked mon.noname-a con 0x7f3f580a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:12.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 10 monclient(hunting): picked mon.noname-c con 0x7f3f5813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:12.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 10 monclient(hunting): picked mon.noname-b con 0x7f3f581430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:12.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f581dabe0 con 0x7f3f580a98f0 2024-03-25T17:43:12.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f581dbab0 con 0x7f3f581430c0 2024-03-25T17:43:12.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f581d5060 con 0x7f3f5813f710 2024-03-25T17:43:12.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:12.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.578+0000 7f3f5e681640 10 monclient(hunting): authenticate will time out at 6195.806517s 2024-03-25T17:43:12.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.596+0000 7f3f5cc1e640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3f581430c0 0x7f3f58138460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56610/0 (socket says 172.21.15.44:56610) 2024-03-25T17:43:12.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.596+0000 7f3f5cc1e640 1 -- 172.21.15.44:0/496001184 learned_addr learned my addr 172.21.15.44:0/496001184 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:12.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.597+0000 7f3f54ff9640 10 client.?.objecter ms_handle_connect 0x7f3f581430c0 2024-03-25T17:43:12.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.597+0000 7f3f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:12.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.603+0000 7f3f54ff9640 10 client.?.objecter ms_handle_connect 0x7f3f5813f710 2024-03-25T17:43:12.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.603+0000 7f3f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:12.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.606+0000 7f3f54ff9640 10 client.?.objecter ms_handle_connect 0x7f3f580a98f0 2024-03-25T17:43:12.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.606+0000 7f3f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:12.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.607+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706323375 0 0) 0x7f3f581dbab0 con 0x7f3f581430c0 2024-03-25T17:43:12.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.607+0000 7f3f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:43:12.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.607+0000 7f3f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:12.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.607+0000 7f3f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:12.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.607+0000 7f3f54ff9640 10 monclient(hunting): my global_id is 4405 2024-03-25T17:43:12.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.607+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f44004eb0 con 0x7f3f581430c0 2024-03-25T17:43:12.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.621+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78743770 0 0) 0x7f3f581dabe0 con 0x7f3f580a98f0 2024-03-25T17:43:12.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.621+0000 7f3f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:43:12.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.621+0000 7f3f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:12.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.621+0000 7f3f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:12.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.621+0000 7f3f54ff9640 10 monclient(hunting): my global_id is 4416 2024-03-25T17:43:12.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.621+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f581dbab0 con 0x7f3f580a98f0 2024-03-25T17:43:12.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.624+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480002054 0 0) 0x7f3f581d5060 con 0x7f3f5813f710 2024-03-25T17:43:12.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.624+0000 7f3f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:43:12.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.624+0000 7f3f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:12.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.624+0000 7f3f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:12.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.624+0000 7f3f54ff9640 10 monclient(hunting): my global_id is 4403 2024-03-25T17:43:12.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.624+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f581dabe0 con 0x7f3f5813f710 2024-03-25T17:43:12.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.628+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4234363672 0 0) 0x7f3f44004eb0 con 0x7f3f581430c0 2024-03-25T17:43:12.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.628+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f581d5060 con 0x7f3f581430c0 2024-03-25T17:43:12.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.628+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f3f50001c90 con 0x7f3f581430c0 2024-03-25T17:43:12.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.628+0000 7f3f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:12.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.628+0000 7f3f54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.628+0000 7f3f54ff9640 10 monclient(hunting): dump: 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:12.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.628+0000 7f3f54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.641+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1934316928 0 0) 0x7f3f581dbab0 con 0x7f3f580a98f0 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.641+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f44004eb0 con 0x7f3f580a98f0 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.641+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f3f48001ae0 con 0x7f3f580a98f0 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.641+0000 7f3f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.641+0000 7f3f54ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.641+0000 7f3f54ff9640 10 monclient(hunting): dump: 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:12.641 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.642 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.642 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:12.642 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:12.642 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:12.642 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:12.642 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:12.642 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:12.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4174356343 0 0) 0x7f3f581d5060 con 0x7f3f581430c0 2024-03-25T17:43:12.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 >> v1:172.21.15.44:6790/0 conn(0x7f3f5813f710 legacy=0x7f3f581d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 >> v1:172.21.15.44:6789/0 conn(0x7f3f580a98f0 legacy=0x7f3f581db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 1 monclient: found mon.b 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f581d51f0 con 0x7f3f581430c0 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 10 monclient: _finish_auth 0 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 5 monclient: authenticate success, global_id 4405 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 10 monclient: _renew_subs 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:12.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f58137fa0 con 0x7f3f581430c0 2024-03-25T17:43:12.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:43:12.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 10 monclient: _renew_subs 2024-03-25T17:43:12.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:12.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f581d43c0 con 0x7f3f581430c0 2024-03-25T17:43:12.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.644+0000 7f3f5e681640 1 librados: init done 2024-03-25T17:43:12.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.646+0000 7f3f5e681640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:12.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.646+0000 7f3f5e681640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:12.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.646+0000 7f3f5e681640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:12.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.646+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f581382a0 con 0x7f3f581430c0 2024-03-25T17:43:12.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.951+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f50003990 con 0x7f3f581430c0 2024-03-25T17:43:12.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.951+0000 7f3f54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:12.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.951+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f3f50003b40 con 0x7f3f581430c0 2024-03-25T17:43:12.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.951+0000 7f3f54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:12.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.951+0000 7f3f54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.951+0000 7f3f54ff9640 10 monclient: dump: 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:12.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.951+0000 7f3f567fc640 4 set_mon_vals no callback set 2024-03-25T17:43:12.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.952+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f3f50003d20 con 0x7f3f581430c0 2024-03-25T17:43:12.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.952+0000 7f3f57fff640 10 monclient: get_auth_request con 0x7f3f4406df20 auth_method 0 2024-03-25T17:43:12.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.953+0000 7f3f54ff9640 10 client.4405.objecter ms_handle_connect 0x7f3f4406df20 2024-03-25T17:43:12.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.958+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f3f50054a00 con 0x7f3f581430c0 2024-03-25T17:43:12.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.958+0000 7f3f54ff9640 10 client.4405.objecter ms_dispatch 0x7f3f58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:43:12.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.958+0000 7f3f54ff9640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:43:12.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.958+0000 7f3f54ff9640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:43:12.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.958+0000 7f3f54ff9640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T17:43:12.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.959+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f50050510 con 0x7f3f581430c0 2024-03-25T17:43:12.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.959+0000 7f3f54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:12.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:12.959+0000 7f3f54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:43:13.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.199+0000 7f3f5e681640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:13.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.199+0000 7f3f5e681640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:13.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.199+0000 7f3f5e681640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:13.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.199+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f580627b0 con 0x7f3f581430c0 2024-03-25T17:43:13.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.255+0000 7f3f54ff9640 1 -- 172.21.15.44:0/496001184 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f3f50052440 con 0x7f3f581430c0 2024-03-25T17:43:13.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.255+0000 7f3f54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:13.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.255+0000 7f3f54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:43:13.256 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:43:13.256 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:43:13.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.257+0000 7f3f5e681640 10 librados: watch_flush enter 2024-03-25T17:43:13.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.257+0000 7f3f5e681640 10 librados: watch_flush exit 2024-03-25T17:43:13.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.257+0000 7f3f5e681640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T17:43:13.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.257+0000 7f3f5e681640 10 client.4405.objecter successfully canceled tick 2024-03-25T17:43:13.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.257+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 >> v1:172.21.15.44:6800/3257043132 conn(0x7f3f4406df20 legacy=0x7f3f440703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:13.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.257+0000 7f3f5e681640 10 monclient: shutdown 2024-03-25T17:43:13.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.257+0000 7f3f5e681640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:13.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.257+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 >> v1:172.21.15.111:6789/0 conn(0x7f3f581430c0 legacy=0x7f3f58138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:13.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.258+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 shutdown_connections 2024-03-25T17:43:13.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.258+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 >> 172.21.15.44:0/496001184 conn(0x7f3f580b7ea0 msgr2=0x7f3f58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:13.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.258+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 shutdown_connections 2024-03-25T17:43:13.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.258+0000 7f3f5e681640 1 -- 172.21.15.44:0/496001184 wait complete. 2024-03-25T17:43:13.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:13.259+0000 7f3f5e681640 1 librados: shutdown 2024-03-25T17:43:13.273 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:43:17.820 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:35.175 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:17.831 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:35.187 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:17.869 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:35.225 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:17.876 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:35.196 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:17.889 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:35.209 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:17.901 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:35.257 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:17.914 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:35.270 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:17.928 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:35.248 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:17.946 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:35.302 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:20.080 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.191+0000 7fe4cb1bc640 10 monclient: get_monmap_and_config 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.191+0000 7fe4cb1bc640 10 monclient: build_initial_monmap 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.191+0000 7fe4cb1bc640 10 monclient: monmap: 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:43:20.192548+0000 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:43:20.192548+0000 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:20.192 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:43:20.193 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:43:20.193 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:20.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.192+0000 7fe4cb1bc640 1 Processor -- start 2024-03-25T17:43:20.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.192+0000 7fe4cb1bc640 1 -- start start 2024-03-25T17:43:20.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.192+0000 7fe4cb1bc640 10 monclient: init 2024-03-25T17:43:20.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:20.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:43:20.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c40b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:20.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c40b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:20.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c414be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:20.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c40af940 con 0x7fe4c40b4d60 2024-03-25T17:43:20.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c40b0a70 con 0x7fe4c40b1730 2024-03-25T17:43:20.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c40b0d50 con 0x7fe4c414be90 2024-03-25T17:43:20.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:20.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.193+0000 7fe4cb1bc640 10 monclient(hunting): authenticate will time out at 6203.420823s 2024-03-25T17:43:20.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.197+0000 7fe4c3fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe4c40b1730 0x7fe4c40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:35990/0 (socket says 172.21.15.44:35990) 2024-03-25T17:43:20.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.197+0000 7fe4c3fff640 1 -- 172.21.15.44:0/3502779653 learned_addr learned my addr 172.21.15.44:0/3502779653 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:20.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.208+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980440272 0 0) 0x7fe4c40b0a70 con 0x7fe4c40b1730 2024-03-25T17:43:20.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.208+0000 7fe4c37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:43:20.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.208+0000 7fe4c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:20.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.208+0000 7fe4c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:20.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.208+0000 7fe4c37fe640 10 monclient(hunting): my global_id is 4408 2024-03-25T17:43:20.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.208+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b4003520 con 0x7fe4c40b1730 2024-03-25T17:43:20.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1523585312 0 0) 0x7fe4b4003520 con 0x7fe4c40b1730 2024-03-25T17:43:20.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 >> v1:172.21.15.44:6790/0 conn(0x7fe4c414be90 legacy=0x7fe4c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:20.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 >> v1:172.21.15.44:6789/0 conn(0x7fe4c40b4d60 legacy=0x7fe4c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:20.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:43:20.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 1 monclient: found mon.noname-c 2024-03-25T17:43:20.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:20.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:43:20.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c40b1030 con 0x7fe4c40b1730 2024-03-25T17:43:20.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 10 monclient: _finish_auth 0 2024-03-25T17:43:20.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:43:20.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe4b0001690 con 0x7fe4c40b1730 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4c37fe640 10 monclient: dump: 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:20.220 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:20.221 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:20.221 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:20.221 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:20.221 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:20.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4cb1bc640 5 monclient: authenticate success, global_id 4408 2024-03-25T17:43:20.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.217+0000 7fe4cb1bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.231+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b00017f0 con 0x7fe4c40b1730 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.231+0000 7fe4c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.231+0000 7fe4c37fe640 1 -- 172.21.15.44:0/3502779653 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe4b0001c40 con 0x7fe4c40b1730 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.231+0000 7fe4c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.231+0000 7fe4c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.231+0000 7fe4c37fe640 10 monclient: dump: 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:20.232 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:20.233 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:20.233 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:20.233 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:20.233 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:20.233 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:20.233 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:20.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.231+0000 7fe4cb1bc640 10 monclient: get_monmap_and_config success 2024-03-25T17:43:20.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.231+0000 7fe4cb1bc640 4 set_mon_vals no callback set 2024-03-25T17:43:20.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.232+0000 7fe4cb1bc640 10 monclient: shutdown 2024-03-25T17:43:20.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.232+0000 7fe4cb1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:20.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.232+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/3502779653 >> v1:172.21.15.111:6789/0 conn(0x7fe4c40b1730 legacy=0x7fe4c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:20.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.232+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/3502779653 shutdown_connections 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.232+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/3502779653 >> 172.21.15.44:0/3502779653 conn(0x7fe4c40617a0 msgr2=0x7fe4c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.232+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/3502779653 shutdown_connections 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.232+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/3502779653 wait complete. 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.233+0000 7fe4cb1bc640 10 monclient: build_initial_monmap 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.233+0000 7fe4cb1bc640 10 monclient: monmap: 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:43:20.236 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.233+0000 7fe4cb1bc640 1 librados: starting msgr at 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.233+0000 7fe4cb1bc640 1 librados: starting objecter 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.233+0000 7fe4cb1bc640 1 Processor -- start 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 1 -- start start 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 1 librados: setting wanted keys 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 1 librados: calling monclient init 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 10 monclient: init 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:43:20.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c40b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:20.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c40b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:20.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c414be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:20.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c4142050 con 0x7fe4c40b1730 2024-03-25T17:43:20.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c41dac50 con 0x7fe4c414be90 2024-03-25T17:43:20.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c4140d00 con 0x7fe4c40b4d60 2024-03-25T17:43:20.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:20.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.234+0000 7fe4cb1bc640 10 monclient(hunting): authenticate will time out at 6203.462585s 2024-03-25T17:43:20.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.246+0000 7fe4c9759640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe4c414be90 0x7fe4c41e2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:36010/0 (socket says 172.21.15.44:36010) 2024-03-25T17:43:20.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.246+0000 7fe4c9759640 1 -- 172.21.15.44:0/683247774 learned_addr learned my addr 172.21.15.44:0/683247774 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:20.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.250+0000 7fe4c17fa640 10 client.?.objecter ms_handle_connect 0x7fe4c414be90 2024-03-25T17:43:20.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.250+0000 7fe4c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:20.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.253+0000 7fe4c17fa640 10 client.?.objecter ms_handle_connect 0x7fe4c40b4d60 2024-03-25T17:43:20.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.253+0000 7fe4c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:20.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.254+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639302725 0 0) 0x7fe4c41dac50 con 0x7fe4c414be90 2024-03-25T17:43:20.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.254+0000 7fe4c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:43:20.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.254+0000 7fe4c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:20.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.254+0000 7fe4c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:20.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.254+0000 7fe4c17fa640 10 monclient(hunting): my global_id is 4411 2024-03-25T17:43:20.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.254+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a8004eb0 con 0x7fe4c414be90 2024-03-25T17:43:20.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.255+0000 7fe4c17fa640 10 client.?.objecter ms_handle_connect 0x7fe4c40b1730 2024-03-25T17:43:20.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.255+0000 7fe4c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.266+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2783286882 0 0) 0x7fe4a8004eb0 con 0x7fe4c414be90 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.266+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4c41dac50 con 0x7fe4c414be90 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.266+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe4bc001c90 con 0x7fe4c414be90 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.266+0000 7fe4c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.266+0000 7fe4c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.266+0000 7fe4c17fa640 10 monclient(hunting): dump: 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:20.267 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:20.268 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:20.268 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:20.268 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:20.268 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:20.268 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:20.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.266+0000 7fe4c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:43:20.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.270+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173455232 0 0) 0x7fe4c4142050 con 0x7fe4c40b1730 2024-03-25T17:43:20.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.270+0000 7fe4c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:43:20.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.270+0000 7fe4c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:20.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.270+0000 7fe4c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:20.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 10 monclient(hunting): my global_id is 4422 2024-03-25T17:43:20.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a8004eb0 con 0x7fe4c40b1730 2024-03-25T17:43:20.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3601876702 0 0) 0x7fe4c41dac50 con 0x7fe4c414be90 2024-03-25T17:43:20.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 >> v1:172.21.15.44:6790/0 conn(0x7fe4c40b4d60 legacy=0x7fe4c41db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:20.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 >> v1:172.21.15.44:6789/0 conn(0x7fe4c40b1730 legacy=0x7fe4c41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:20.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:43:20.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 1 monclient: found mon.b 2024-03-25T17:43:20.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c41423e0 con 0x7fe4c414be90 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 10 monclient: _finish_auth 0 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 5 monclient: authenticate success, global_id 4411 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 10 monclient: _renew_subs 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c413b470 con 0x7fe4c414be90 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:43:20.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 10 monclient: _renew_subs 2024-03-25T17:43:20.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:20.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c41dba40 con 0x7fe4c414be90 2024-03-25T17:43:20.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.271+0000 7fe4cb1bc640 1 librados: init done 2024-03-25T17:43:20.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.272+0000 7fe4cb1bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:20.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.272+0000 7fe4cb1bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:20.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.272+0000 7fe4cb1bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:20.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.272+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c41401f0 con 0x7fe4c414be90 2024-03-25T17:43:20.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.551+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4bc003990 con 0x7fe4c414be90 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.551+0000 7fe4c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.551+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe4bc003d20 con 0x7fe4c414be90 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.551+0000 7fe4c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.551+0000 7fe4c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.551+0000 7fe4c17fa640 10 monclient: dump: 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:20.552 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:20.553 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:20.553 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:20.553 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:20.553 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:20.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.551+0000 7fe4c2ffd640 4 set_mon_vals no callback set 2024-03-25T17:43:20.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.552+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fe4bc003f40 con 0x7fe4c414be90 2024-03-25T17:43:20.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.552+0000 7fe4c8f58640 10 monclient: get_auth_request con 0x7fe4a806ddf0 auth_method 0 2024-03-25T17:43:20.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.552+0000 7fe4c17fa640 10 client.4411.objecter ms_handle_connect 0x7fe4a806ddf0 2024-03-25T17:43:20.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.557+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fe4bc054a30 con 0x7fe4c414be90 2024-03-25T17:43:20.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.557+0000 7fe4c17fa640 10 client.4411.objecter ms_dispatch 0x7fe4c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:43:20.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.557+0000 7fe4c17fa640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:43:20.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.557+0000 7fe4c17fa640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:43:20.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.557+0000 7fe4c17fa640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T17:43:20.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.559+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4bc050540 con 0x7fe4c414be90 2024-03-25T17:43:20.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.559+0000 7fe4c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:20.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.559+0000 7fe4c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:43:20.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.816+0000 7fe4cb1bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:20.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.816+0000 7fe4cb1bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:20.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.816+0000 7fe4cb1bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:20.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.816+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4c4140e90 con 0x7fe4c414be90 2024-03-25T17:43:20.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.851+0000 7fe4c17fa640 1 -- 172.21.15.44:0/683247774 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fe4bc052470 con 0x7fe4c414be90 2024-03-25T17:43:20.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.851+0000 7fe4c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:20.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.851+0000 7fe4c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:43:20.852 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:43:20.853 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 10 librados: watch_flush enter 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 10 librados: watch_flush exit 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 10 client.4411.objecter successfully canceled tick 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 >> v1:172.21.15.44:6800/3257043132 conn(0x7fe4a806ddf0 legacy=0x7fe4a80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 10 monclient: shutdown 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 >> v1:172.21.15.111:6789/0 conn(0x7fe4c414be90 legacy=0x7fe4c41e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:20.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 shutdown_connections 2024-03-25T17:43:20.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 >> 172.21.15.44:0/683247774 conn(0x7fe4c40617a0 msgr2=0x7fe4c40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:20.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 shutdown_connections 2024-03-25T17:43:20.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 1 -- 172.21.15.44:0/683247774 wait complete. 2024-03-25T17:43:20.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:20.854+0000 7fe4cb1bc640 1 librados: shutdown 2024-03-25T17:43:20.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:43:27.675 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.788+0000 7fa38293b640 10 monclient: get_monmap_and_config 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.788+0000 7fa38293b640 10 monclient: build_initial_monmap 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.788+0000 7fa38293b640 10 monclient: monmap: 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:43:27.789744+0000 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:43:27.789744+0000 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:27.789 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:27.790 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:27.790 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:43:27.790 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:43:27.790 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:27.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.789+0000 7fa38293b640 1 Processor -- start 2024-03-25T17:43:27.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.789+0000 7fa38293b640 1 -- start start 2024-03-25T17:43:27.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.789+0000 7fa38293b640 10 monclient: init 2024-03-25T17:43:27.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:27.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:43:27.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 10 monclient(hunting): picked mon.noname-b con 0x7fa37c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:27.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 10 monclient(hunting): picked mon.noname-a con 0x7fa37c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:27.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 10 monclient(hunting): picked mon.noname-c con 0x7fa37c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:27.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c142520 con 0x7fa37c13f710 2024-03-25T17:43:27.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0b28a0 con 0x7fa37c1430c0 2024-03-25T17:43:27.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0b2a30 con 0x7fa37c0a98f0 2024-03-25T17:43:27.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:27.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.790+0000 7fa38293b640 10 monclient(hunting): authenticate will time out at 6211.018174s 2024-03-25T17:43:27.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.804+0000 7fa37bfff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fa37c0a98f0 0x7fa37c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:59496/0 (socket says 172.21.15.44:59496) 2024-03-25T17:43:27.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.804+0000 7fa37bfff640 1 -- 172.21.15.44:0/3329665924 learned_addr learned my addr 172.21.15.44:0/3329665924 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:27.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.820+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894737966 0 0) 0x7fa37c0b2a30 con 0x7fa37c0a98f0 2024-03-25T17:43:27.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.820+0000 7fa37affd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:27.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.820+0000 7fa37affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:27.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.820+0000 7fa37affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:27.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.820+0000 7fa37affd640 10 monclient(hunting): my global_id is 4412 2024-03-25T17:43:27.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.820+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa364003520 con 0x7fa37c0a98f0 2024-03-25T17:43:27.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.825+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029613908 0 0) 0x7fa37c0b28a0 con 0x7fa37c1430c0 2024-03-25T17:43:27.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.825+0000 7fa37affd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:27.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.825+0000 7fa37affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:27.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.825+0000 7fa37affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:27.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.825+0000 7fa37affd640 10 monclient(hunting): my global_id is 4414 2024-03-25T17:43:27.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.825+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa37c0b2a30 con 0x7fa37c1430c0 2024-03-25T17:43:27.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.837+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986094927 0 0) 0x7fa37c142520 con 0x7fa37c13f710 2024-03-25T17:43:27.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.837+0000 7fa37affd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:27.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.837+0000 7fa37affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:27.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.837+0000 7fa37affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:27.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.837+0000 7fa37affd640 10 monclient(hunting): my global_id is 4425 2024-03-25T17:43:27.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.837+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa37c0b28a0 con 0x7fa37c13f710 2024-03-25T17:43:27.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1427582940 0 0) 0x7fa37c0b28a0 con 0x7fa37c13f710 2024-03-25T17:43:27.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 >> v1:172.21.15.44:6790/0 conn(0x7fa37c0a98f0 legacy=0x7fa37c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:27.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 >> v1:172.21.15.111:6789/0 conn(0x7fa37c1430c0 legacy=0x7fa37c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:27.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 10 monclient: _finish_hunting 0 2024-03-25T17:43:27.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 1 monclient: found mon.noname-a 2024-03-25T17:43:27.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:27.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:43:27.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0b44c0 con 0x7fa37c13f710 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 10 monclient: _finish_auth 0 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 10 monclient: _check_auth_tickets 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa3700015b0 con 0x7fa37c13f710 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa37affd640 10 monclient: dump: 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:27.846 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:27.847 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:27.847 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:27.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa38293b640 5 monclient: authenticate success, global_id 4425 2024-03-25T17:43:27.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.844+0000 7fa38293b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.878+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa370001710 con 0x7fa37c13f710 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.878+0000 7fa37affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.878+0000 7fa37affd640 1 -- 172.21.15.44:0/3329665924 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa370001b60 con 0x7fa37c13f710 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.878+0000 7fa37affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.878+0000 7fa37affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.878+0000 7fa37affd640 10 monclient: dump: 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:27.879 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:27.880 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:27.880 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:27.880 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:27.880 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:27.880 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:27.880 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:27.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.878+0000 7fa38293b640 10 monclient: get_monmap_and_config success 2024-03-25T17:43:27.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.878+0000 7fa38293b640 4 set_mon_vals no callback set 2024-03-25T17:43:27.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.879+0000 7fa38293b640 10 monclient: shutdown 2024-03-25T17:43:27.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.879+0000 7fa38293b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:27.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.879+0000 7fa38293b640 1 -- 172.21.15.44:0/3329665924 >> v1:172.21.15.44:6789/0 conn(0x7fa37c13f710 legacy=0x7fa37c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:27.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.879+0000 7fa38293b640 1 -- 172.21.15.44:0/3329665924 shutdown_connections 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.879+0000 7fa38293b640 1 -- 172.21.15.44:0/3329665924 >> 172.21.15.44:0/3329665924 conn(0x7fa37c0b7ea0 msgr2=0x7fa37c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.879+0000 7fa38293b640 1 -- 172.21.15.44:0/3329665924 shutdown_connections 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.879+0000 7fa38293b640 1 -- 172.21.15.44:0/3329665924 wait complete. 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.880+0000 7fa38293b640 10 monclient: build_initial_monmap 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.880+0000 7fa38293b640 10 monclient: monmap: 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:43:27.883 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.880+0000 7fa38293b640 1 librados: starting msgr at 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.880+0000 7fa38293b640 1 librados: starting objecter 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.880+0000 7fa38293b640 1 Processor -- start 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.881+0000 7fa38293b640 1 -- start start 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.881+0000 7fa38293b640 1 librados: setting wanted keys 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.881+0000 7fa38293b640 1 librados: calling monclient init 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.881+0000 7fa38293b640 10 monclient: init 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.881+0000 7fa38293b640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.881+0000 7fa38293b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.881+0000 7fa38293b640 10 monclient(hunting): picked mon.noname-b con 0x7fa37c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.882+0000 7fa38293b640 10 monclient(hunting): picked mon.noname-a con 0x7fa37c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.882+0000 7fa38293b640 10 monclient(hunting): picked mon.noname-c con 0x7fa37c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.882+0000 7fa38293b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0abd30 con 0x7fa37c13f710 2024-03-25T17:43:27.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.882+0000 7fa38293b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c1d51f0 con 0x7fa37c0a98f0 2024-03-25T17:43:27.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.882+0000 7fa38293b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c1d77a0 con 0x7fa37c1430c0 2024-03-25T17:43:27.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.882+0000 7fa38293b640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:27.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.882+0000 7fa38293b640 10 monclient(hunting): authenticate will time out at 6211.109785s 2024-03-25T17:43:27.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.885+0000 7fa37bfff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa37c0a98f0 0x7fa37c138740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:40948/0 (socket says 172.21.15.44:40948) 2024-03-25T17:43:27.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.885+0000 7fa37bfff640 1 -- 172.21.15.44:0/560494182 learned_addr learned my addr 172.21.15.44:0/560494182 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:27.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.886+0000 7fa378ff9640 10 client.?.objecter ms_handle_connect 0x7fa37c0a98f0 2024-03-25T17:43:27.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.886+0000 7fa378ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:27.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.899+0000 7fa378ff9640 10 client.?.objecter ms_handle_connect 0x7fa37c13f710 2024-03-25T17:43:27.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.899+0000 7fa378ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:27.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.902+0000 7fa378ff9640 10 client.?.objecter ms_handle_connect 0x7fa37c1430c0 2024-03-25T17:43:27.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.902+0000 7fa378ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:27.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.902+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995932737 0 0) 0x7fa37c1d51f0 con 0x7fa37c0a98f0 2024-03-25T17:43:27.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.902+0000 7fa378ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:43:27.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.902+0000 7fa378ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:27.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.902+0000 7fa378ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:27.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.902+0000 7fa378ff9640 10 monclient(hunting): my global_id is 4417 2024-03-25T17:43:27.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.902+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa368004eb0 con 0x7fa37c0a98f0 2024-03-25T17:43:27.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.909+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694941774 0 0) 0x7fa37c1d77a0 con 0x7fa37c1430c0 2024-03-25T17:43:27.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.909+0000 7fa378ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:43:27.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.909+0000 7fa378ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:27.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.909+0000 7fa378ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:27.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.909+0000 7fa378ff9640 10 monclient(hunting): my global_id is 4415 2024-03-25T17:43:27.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.909+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa37c1d51f0 con 0x7fa37c1430c0 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.914+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716310382 0 0) 0x7fa368004eb0 con 0x7fa37c0a98f0 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.914+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa37c1d77a0 con 0x7fa37c0a98f0 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.914+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa36c001950 con 0x7fa37c0a98f0 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.914+0000 7fa378ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.914+0000 7fa378ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.914+0000 7fa378ff9640 10 monclient(hunting): dump: 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:27.915 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:27.916 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:27.916 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:27.916 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:27.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.914+0000 7fa378ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:43:27.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.919+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3664446217 0 0) 0x7fa37c1d51f0 con 0x7fa37c1430c0 2024-03-25T17:43:27.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.919+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa368004eb0 con 0x7fa37c1430c0 2024-03-25T17:43:27.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.919+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229452163 0 0) 0x7fa37c0abd30 con 0x7fa37c13f710 2024-03-25T17:43:27.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.919+0000 7fa378ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:43:27.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.919+0000 7fa378ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:27.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.919+0000 7fa378ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:27.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.919+0000 7fa378ff9640 10 monclient(hunting): my global_id is 4428 2024-03-25T17:43:27.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.919+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa37c1d51f0 con 0x7fa37c13f710 2024-03-25T17:43:27.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.921+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa374001bf0 con 0x7fa37c1430c0 2024-03-25T17:43:27.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.921+0000 7fa378ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:27.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.921+0000 7fa378ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:43:27.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.921+0000 7fa378ff9640 10 monclient(hunting): dump: 2024-03-25T17:43:27.921 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:27.921 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:27.921 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:27.921 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:27.922 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:27.922 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:27.922 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:27.922 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:27.922 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:27.922 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:27.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.933+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2950776600 0 0) 0x7fa37c1d77a0 con 0x7fa37c0a98f0 2024-03-25T17:43:27.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.933+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 >> v1:172.21.15.44:6790/0 conn(0x7fa37c1430c0 legacy=0x7fa37c1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:27.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.933+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 >> v1:172.21.15.44:6789/0 conn(0x7fa37c13f710 legacy=0x7fa37c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:27.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa378ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:43:27.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa378ff9640 1 monclient: found mon.b 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa378ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa378ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c138e50 con 0x7fa37c0a98f0 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa378ff9640 10 monclient: _finish_auth 0 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa378ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa378ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 5 monclient: authenticate success, global_id 4417 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 10 monclient: _renew_subs 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37c1d83d0 con 0x7fa37c0a98f0 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 10 monclient: _renew_subs 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37c1381b0 con 0x7fa37c0a98f0 2024-03-25T17:43:27.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.934+0000 7fa38293b640 1 librados: init done 2024-03-25T17:43:27.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.935+0000 7fa38293b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:27.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.935+0000 7fa38293b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:27.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.935+0000 7fa38293b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:27.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:27.935+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37c1d7600 con 0x7fa37c0a98f0 2024-03-25T17:43:28.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.204+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa36c0047e0 con 0x7fa37c0a98f0 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.204+0000 7fa378ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.204+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa36c004950 con 0x7fa37c0a98f0 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.204+0000 7fa378ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.204+0000 7fa378ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.204+0000 7fa378ff9640 10 monclient: dump: 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:28.205 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:28.206 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:28.206 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:28.206 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:28.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.204+0000 7fa37a7fc640 4 set_mon_vals no callback set 2024-03-25T17:43:28.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.205+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fa36c01a920 con 0x7fa37c0a98f0 2024-03-25T17:43:28.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.205+0000 7fa37b7fe640 10 monclient: get_auth_request con 0x7fa36806df30 auth_method 0 2024-03-25T17:43:28.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.206+0000 7fa378ff9640 10 client.4417.objecter ms_handle_connect 0x7fa36806df30 2024-03-25T17:43:28.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.213+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fa36c0546c0 con 0x7fa37c0a98f0 2024-03-25T17:43:28.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.213+0000 7fa378ff9640 10 client.4417.objecter ms_dispatch 0x7fa37c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:43:28.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.213+0000 7fa378ff9640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:43:28.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.213+0000 7fa378ff9640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:43:28.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.213+0000 7fa378ff9640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T17:43:28.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.214+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa36c050000 con 0x7fa37c0a98f0 2024-03-25T17:43:28.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.214+0000 7fa378ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:28.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.214+0000 7fa378ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:43:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.429+0000 7fa38293b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.429+0000 7fa38293b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.429+0000 7fa38293b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.429+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa37c1d8560 con 0x7fa37c0a98f0 2024-03-25T17:43:28.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.497+0000 7fa378ff9640 1 -- 172.21.15.44:0/560494182 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fa36c052100 con 0x7fa37c0a98f0 2024-03-25T17:43:28.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.497+0000 7fa378ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:28.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.497+0000 7fa378ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:43:28.498 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:43:28.498 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:43:28.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 10 librados: watch_flush enter 2024-03-25T17:43:28.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 10 librados: watch_flush exit 2024-03-25T17:43:28.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T17:43:28.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 10 client.4417.objecter successfully canceled tick 2024-03-25T17:43:28.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 >> v1:172.21.15.44:6800/3257043132 conn(0x7fa36806df30 legacy=0x7fa3680703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:28.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 10 monclient: shutdown 2024-03-25T17:43:28.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:28.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 >> v1:172.21.15.111:6789/0 conn(0x7fa37c0a98f0 legacy=0x7fa37c138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:28.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 shutdown_connections 2024-03-25T17:43:28.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.499+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 >> 172.21.15.44:0/560494182 conn(0x7fa37c0b7ea0 msgr2=0x7fa37c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:28.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.500+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 shutdown_connections 2024-03-25T17:43:28.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.500+0000 7fa38293b640 1 -- 172.21.15.44:0/560494182 wait complete. 2024-03-25T17:43:28.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:28.500+0000 7fa38293b640 1 librados: shutdown 2024-03-25T17:43:28.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:43:35.317 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:43:35.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.435+0000 7f64e7504640 10 monclient: get_monmap_and_config 2024-03-25T17:43:35.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.435+0000 7f64e7504640 10 monclient: build_initial_monmap 2024-03-25T17:43:35.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.435+0000 7f64e7504640 10 monclient: monmap: 2024-03-25T17:43:35.435 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:35.435 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:35.436 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:43:35.436200+0000 2024-03-25T17:43:35.436 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:43:35.436200+0000 2024-03-25T17:43:35.436 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:35.436 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:35.436 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:35.436 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:43:35.436 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:43:35.436 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:35.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.435+0000 7f64e7504640 1 Processor -- start 2024-03-25T17:43:35.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.436+0000 7f64e7504640 1 -- start start 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.436+0000 7f64e7504640 10 monclient: init 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.436+0000 7f64e7504640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.436+0000 7f64e7504640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.437+0000 7f64e7504640 10 monclient(hunting): picked mon.noname-b con 0x7f64e006ab80 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.437+0000 7f64e7504640 10 monclient(hunting): picked mon.noname-a con 0x7f64e006af60 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.437+0000 7f64e7504640 10 monclient(hunting): picked mon.noname-c con 0x7f64e013f990 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.437+0000 7f64e7504640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e00afa60 con 0x7f64e006af60 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.437+0000 7f64e7504640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e00afce0 con 0x7f64e013f990 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.437+0000 7f64e7504640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e00b1150 con 0x7f64e006ab80 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.437+0000 7f64e7504640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:35.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.437+0000 7f64e7504640 10 monclient(hunting): authenticate will time out at 6218.664980s 2024-03-25T17:43:35.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.449+0000 7f64e5aa1640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f64e013f990 0x7f64e014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:34762/0 (socket says 172.21.15.44:34762) 2024-03-25T17:43:35.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.449+0000 7f64e5aa1640 1 -- 172.21.15.44:0/2658324215 learned_addr learned my addr 172.21.15.44:0/2658324215 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:35.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.454+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754695265 0 0) 0x7f64e00afce0 con 0x7f64e013f990 2024-03-25T17:43:35.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.454+0000 7f64d7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:43:35.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.454+0000 7f64d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:35.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.454+0000 7f64d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:35.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.454+0000 7f64d7fff640 10 monclient(hunting): my global_id is 4420 2024-03-25T17:43:35.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.454+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64cc003520 con 0x7f64e013f990 2024-03-25T17:43:35.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.464+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418379708 0 0) 0x7f64e00afa60 con 0x7f64e006af60 2024-03-25T17:43:35.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.464+0000 7f64d7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:43:35.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.464+0000 7f64d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:35.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.464+0000 7f64d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:35.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.464+0000 7f64d7fff640 10 monclient(hunting): my global_id is 4431 2024-03-25T17:43:35.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.464+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e00afce0 con 0x7f64e006af60 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2086787655 0 0) 0x7f64cc003520 con 0x7f64e013f990 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 >> v1:172.21.15.44:6790/0 conn(0x7f64e006ab80 legacy=0x7f64e00724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 >> v1:172.21.15.44:6789/0 conn(0x7f64e006af60 legacy=0x7f64e01458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 1 monclient: found mon.noname-c 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e00aff20 con 0x7f64e013f990 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 10 monclient: _finish_auth 0 2024-03-25T17:43:35.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64e7504640 5 monclient: authenticate success, global_id 4420 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.466+0000 7f64e7504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.467+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f64d80019d0 con 0x7f64e013f990 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.467+0000 7f64d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.467+0000 7f64d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.467+0000 7f64d7fff640 10 monclient: dump: 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:35.468 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:35.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.467+0000 7f64e7504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64d8001c20 con 0x7f64e013f990 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64d7fff640 1 -- 172.21.15.44:0/2658324215 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f64d8001fb0 con 0x7f64e013f990 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64e7504640 10 monclient: get_monmap_and_config success 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64e7504640 4 set_mon_vals no callback set 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64d7fff640 10 monclient: dump: 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:35.472 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:35.473 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:35.473 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:35.473 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:35.473 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:35.473 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:35.473 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:35.473 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:35.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64e7504640 10 monclient: shutdown 2024-03-25T17:43:35.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64e7504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:35.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.472+0000 7f64e7504640 1 -- 172.21.15.44:0/2658324215 >> v1:172.21.15.111:6789/0 conn(0x7f64e013f990 legacy=0x7f64e014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:35.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.473+0000 7f64e7504640 1 -- 172.21.15.44:0/2658324215 shutdown_connections 2024-03-25T17:43:35.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.473+0000 7f64e7504640 1 -- 172.21.15.44:0/2658324215 >> 172.21.15.44:0/2658324215 conn(0x7f64e006c830 msgr2=0x7f64e00b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:35.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.473+0000 7f64e7504640 1 -- 172.21.15.44:0/2658324215 shutdown_connections 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.473+0000 7f64e7504640 1 -- 172.21.15.44:0/2658324215 wait complete. 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.473+0000 7f64e7504640 10 monclient: build_initial_monmap 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.473+0000 7f64e7504640 10 monclient: monmap: 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:43:35.475 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:35.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 1 librados: starting msgr at 2024-03-25T17:43:35.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 1 librados: starting objecter 2024-03-25T17:43:35.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 1 Processor -- start 2024-03-25T17:43:35.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 1 -- start start 2024-03-25T17:43:35.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 1 librados: setting wanted keys 2024-03-25T17:43:35.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 1 librados: calling monclient init 2024-03-25T17:43:35.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 10 monclient: init 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.474+0000 7f64e7504640 10 monclient(hunting): picked mon.noname-b con 0x7f64e006ab80 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.475+0000 7f64e7504640 10 monclient(hunting): picked mon.noname-c con 0x7f64e013f990 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.475+0000 7f64e7504640 10 monclient(hunting): picked mon.noname-a con 0x7f64e0142dc0 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.475+0000 7f64e7504640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e0142740 con 0x7f64e0142dc0 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.475+0000 7f64e7504640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e01419d0 con 0x7f64e006ab80 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.475+0000 7f64e7504640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e0142190 con 0x7f64e013f990 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.475+0000 7f64e7504640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:35.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.475+0000 7f64e7504640 10 monclient(hunting): authenticate will time out at 6218.702806s 2024-03-25T17:43:35.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.481+0000 7f64e52a0640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f64e006ab80 0x7f64e0139c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:34816/0 (socket says 172.21.15.44:34816) 2024-03-25T17:43:35.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.481+0000 7f64e52a0640 1 -- 172.21.15.44:0/3096784626 learned_addr learned my addr 172.21.15.44:0/3096784626 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:35.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.485+0000 7f64d5ffb640 10 client.?.objecter ms_handle_connect 0x7f64e006ab80 2024-03-25T17:43:35.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.485+0000 7f64d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:35.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.490+0000 7f64d5ffb640 10 client.?.objecter ms_handle_connect 0x7f64e013f990 2024-03-25T17:43:35.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.490+0000 7f64d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:35.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.491+0000 7f64d5ffb640 10 client.?.objecter ms_handle_connect 0x7f64e0142dc0 2024-03-25T17:43:35.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.491+0000 7f64d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:35.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.493+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010561285 0 0) 0x7f64e01419d0 con 0x7f64e006ab80 2024-03-25T17:43:35.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.493+0000 7f64d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:43:35.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.493+0000 7f64d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:35.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.493+0000 7f64d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:35.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.493+0000 7f64d5ffb640 10 monclient(hunting): my global_id is 4423 2024-03-25T17:43:35.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.493+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c4004eb0 con 0x7f64e006ab80 2024-03-25T17:43:35.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.499+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799685308 0 0) 0x7f64e0142190 con 0x7f64e013f990 2024-03-25T17:43:35.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.499+0000 7f64d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:43:35.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.499+0000 7f64d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:35.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.499+0000 7f64d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:35.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.499+0000 7f64d5ffb640 10 monclient(hunting): my global_id is 4421 2024-03-25T17:43:35.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.499+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e01419d0 con 0x7f64e013f990 2024-03-25T17:43:35.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.500+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269685793 0 0) 0x7f64e0142740 con 0x7f64e0142dc0 2024-03-25T17:43:35.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.500+0000 7f64d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:43:35.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.500+0000 7f64d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:35.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.500+0000 7f64d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:35.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.500+0000 7f64d5ffb640 10 monclient(hunting): my global_id is 4434 2024-03-25T17:43:35.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.500+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e0142190 con 0x7f64e0142dc0 2024-03-25T17:43:35.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.505+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209423790 0 0) 0x7f64c4004eb0 con 0x7f64e006ab80 2024-03-25T17:43:35.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.505+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64e0142740 con 0x7f64e006ab80 2024-03-25T17:43:35.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.505+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f64dc0019d0 con 0x7f64e006ab80 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.505+0000 7f64d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.505+0000 7f64d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.505+0000 7f64d5ffb640 10 monclient(hunting): dump: 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.505+0000 7f64d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1129122418 0 0) 0x7f64e0142740 con 0x7f64e006ab80 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 >> v1:172.21.15.44:6790/0 conn(0x7f64e013f990 legacy=0x7f64e01dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 >> v1:172.21.15.44:6789/0 conn(0x7f64e0142dc0 legacy=0x7f64e01db460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 1 monclient: found mon.b 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e0142320 con 0x7f64e006ab80 2024-03-25T17:43:35.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 10 monclient: _finish_auth 0 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64e7504640 5 monclient: authenticate success, global_id 4423 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64e7504640 10 monclient: _renew_subs 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64e7504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.518+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e0144c20 con 0x7f64e006ab80 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.519+0000 7f64e7504640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.519+0000 7f64e7504640 10 monclient: _renew_subs 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.519+0000 7f64e7504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.519+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e013a3a0 con 0x7f64e006ab80 2024-03-25T17:43:35.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.519+0000 7f64e7504640 1 librados: init done 2024-03-25T17:43:35.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.520+0000 7f64e7504640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:35.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.520+0000 7f64e7504640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:35.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.520+0000 7f64e7504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:35.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.520+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e00627b0 con 0x7f64e006ab80 2024-03-25T17:43:35.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.522+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64dc0035f0 con 0x7f64e006ab80 2024-03-25T17:43:35.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.522+0000 7f64d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:35.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.523+0000 7f64d77fe640 4 set_mon_vals no callback set 2024-03-25T17:43:35.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.786+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f64dc003a00 con 0x7f64e006ab80 2024-03-25T17:43:35.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.787+0000 7f64d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:35.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.787+0000 7f64d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:35.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.787+0000 7f64d5ffb640 10 monclient: dump: 2024-03-25T17:43:35.787 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:35.787 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:35.787 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:35.788 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:35.788 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:35.788 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:35.788 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:35.788 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:35.788 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:35.788 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:35.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.787+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f64dc003be0 con 0x7f64e006ab80 2024-03-25T17:43:35.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.788+0000 7f64e4a9f640 10 monclient: get_auth_request con 0x7f64c406ddd0 auth_method 0 2024-03-25T17:43:35.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.788+0000 7f64d5ffb640 10 client.4423.objecter ms_handle_connect 0x7f64c406ddd0 2024-03-25T17:43:35.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.793+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f64dc054800 con 0x7f64e006ab80 2024-03-25T17:43:35.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.793+0000 7f64d5ffb640 10 client.4423.objecter ms_dispatch 0x7f64e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:43:35.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.793+0000 7f64d5ffb640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:43:35.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.793+0000 7f64d5ffb640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:43:35.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.793+0000 7f64d5ffb640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T17:43:35.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.796+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64dc050310 con 0x7f64e006ab80 2024-03-25T17:43:35.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.796+0000 7f64d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:35.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:35.796+0000 7f64d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:43:36.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.045+0000 7f64e7504640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:36.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.045+0000 7f64e7504640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:36.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.045+0000 7f64e7504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:43:36.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.045+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64e0070790 con 0x7f64e006ab80 2024-03-25T17:43:36.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.169+0000 7f64d5ffb640 1 -- 172.21.15.44:0/3096784626 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f64dc052240 con 0x7f64e006ab80 2024-03-25T17:43:36.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.169+0000 7f64d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:36.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.169+0000 7f64d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:43:36.171 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:43:36.171 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:43:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 10 librados: watch_flush enter 2024-03-25T17:43:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 10 librados: watch_flush exit 2024-03-25T17:43:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T17:43:36.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 10 client.4423.objecter successfully canceled tick 2024-03-25T17:43:36.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 >> v1:172.21.15.44:6800/3257043132 conn(0x7f64c406ddd0 legacy=0x7f64c4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:36.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 10 monclient: shutdown 2024-03-25T17:43:36.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:36.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 >> v1:172.21.15.111:6789/0 conn(0x7f64e006ab80 legacy=0x7f64e0139c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:36.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 shutdown_connections 2024-03-25T17:43:36.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 >> 172.21.15.44:0/3096784626 conn(0x7f64e006c830 msgr2=0x7f64e00b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:36.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 shutdown_connections 2024-03-25T17:43:36.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 1 -- 172.21.15.44:0/3096784626 wait complete. 2024-03-25T17:43:36.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:36.172+0000 7f64e7504640 1 librados: shutdown 2024-03-25T17:43:36.184 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:43:37.832 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:55.188 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:37.887 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:55.242 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:37.929 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:55.285 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:37.947 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:55.303 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:37.948 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:55.268 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:37.955 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:55.311 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:37.956 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:55.276 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:37.984 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:55.304 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:37.987 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:55.343 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:42.991 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:43:43.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.103+0000 7f1a8390e640 10 monclient: get_monmap_and_config 2024-03-25T17:43:43.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.103+0000 7f1a8390e640 10 monclient: build_initial_monmap 2024-03-25T17:43:43.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.103+0000 7f1a8390e640 10 monclient: monmap: 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:43:43.104066+0000 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:43:43.104066+0000 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:43:43.104 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.104+0000 7f1a8390e640 1 Processor -- start 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.104+0000 7f1a8390e640 1 -- start start 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.104+0000 7f1a8390e640 10 monclient: init 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 10 monclient(hunting): picked mon.noname-a con 0x7f1a7c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c142520 con 0x7f1a7c1430c0 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c0b28a0 con 0x7f1a7c13f710 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c0b2a30 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:43.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.105+0000 7f1a8390e640 10 monclient(hunting): authenticate will time out at 6226.333238s 2024-03-25T17:43:43.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.118+0000 7f1a80ea9640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1a7c13f710 0x7f1a7c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:41050/0 (socket says 172.21.15.44:41050) 2024-03-25T17:43:43.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.119+0000 7f1a80ea9640 1 -- 172.21.15.44:0/2519561424 learned_addr learned my addr 172.21.15.44:0/2519561424 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:43.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.130+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269521121 0 0) 0x7f1a7c0b28a0 con 0x7f1a7c13f710 2024-03-25T17:43:43.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.131+0000 7f1a73fff640 10 monclient(hunting): _negotiate 2024-03-25T17:43:43.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.131+0000 7f1a73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:43.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.131+0000 7f1a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:43.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.131+0000 7f1a73fff640 10 monclient(hunting): my global_id is 4426 2024-03-25T17:43:43.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.131+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a68003520 con 0x7f1a7c13f710 2024-03-25T17:43:43.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.137+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229260113 0 0) 0x7f1a7c0b2a30 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.137+0000 7f1a73fff640 10 monclient(hunting): _negotiate 2024-03-25T17:43:43.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.137+0000 7f1a73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:43.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.137+0000 7f1a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:43.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.137+0000 7f1a73fff640 10 monclient(hunting): my global_id is 4424 2024-03-25T17:43:43.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.137+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c0b28a0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.139+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912214616 0 0) 0x7f1a7c142520 con 0x7f1a7c1430c0 2024-03-25T17:43:43.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.139+0000 7f1a73fff640 10 monclient(hunting): _negotiate 2024-03-25T17:43:43.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.139+0000 7f1a73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:43.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.139+0000 7f1a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:43.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.139+0000 7f1a73fff640 10 monclient(hunting): my global_id is 4437 2024-03-25T17:43:43.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.139+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c0b2a30 con 0x7f1a7c1430c0 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3827483139 0 0) 0x7f1a7c0b28a0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 >> v1:172.21.15.111:6789/0 conn(0x7f1a7c13f710 legacy=0x7f1a7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 >> v1:172.21.15.44:6789/0 conn(0x7f1a7c1430c0 legacy=0x7f1a7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 10 monclient: _finish_hunting 0 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 1 monclient: found mon.noname-b 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c0b44c0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 10 monclient: _finish_auth 0 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 10 monclient: _check_auth_tickets 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1a78001670 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a8390e640 5 monclient: authenticate success, global_id 4424 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a8390e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a73fff640 10 monclient: dump: 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:43.151 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:43.152 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:43.152 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:43.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.149+0000 7f1a8390e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:43.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.172+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a780017d0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.172+0000 7f1a73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:43.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.172+0000 7f1a73fff640 1 -- 172.21.15.44:0/2519561424 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1a78001be0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.172+0000 7f1a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:43.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.172+0000 7f1a73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.172+0000 7f1a73fff640 10 monclient: dump: 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.172+0000 7f1a8390e640 10 monclient: get_monmap_and_config success 2024-03-25T17:43:43.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.172+0000 7f1a8390e640 4 set_mon_vals no callback set 2024-03-25T17:43:43.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.173+0000 7f1a8390e640 10 monclient: shutdown 2024-03-25T17:43:43.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.173+0000 7f1a8390e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.173+0000 7f1a8390e640 1 -- 172.21.15.44:0/2519561424 >> v1:172.21.15.44:6790/0 conn(0x7f1a7c0a98f0 legacy=0x7f1a7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.173+0000 7f1a8390e640 1 -- 172.21.15.44:0/2519561424 shutdown_connections 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.173+0000 7f1a8390e640 1 -- 172.21.15.44:0/2519561424 >> 172.21.15.44:0/2519561424 conn(0x7f1a7c0b7ea0 msgr2=0x7f1a7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.173+0000 7f1a8390e640 1 -- 172.21.15.44:0/2519561424 shutdown_connections 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.173+0000 7f1a8390e640 1 -- 172.21.15.44:0/2519561424 wait complete. 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 10 monclient: build_initial_monmap 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 10 monclient: monmap: 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:43.176 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 1 librados: starting msgr at 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 1 librados: starting objecter 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 1 Processor -- start 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 1 -- start start 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 1 librados: setting wanted keys 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 1 librados: calling monclient init 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.174+0000 7f1a8390e640 10 monclient: init 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:43.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 10 monclient(hunting): picked mon.noname-a con 0x7f1a7c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:43.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c136e60 con 0x7f1a7c1430c0 2024-03-25T17:43:43.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c1381a0 con 0x7f1a7c13f710 2024-03-25T17:43:43.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c1370b0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:43.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.175+0000 7f1a8390e640 10 monclient(hunting): authenticate will time out at 6226.403036s 2024-03-25T17:43:43.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.188+0000 7f1a816aa640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f1a7c0a98f0 0x7f1a7c137a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:48112/0 (socket says 172.21.15.44:48112) 2024-03-25T17:43:43.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.188+0000 7f1a816aa640 1 -- 172.21.15.44:0/33594174 learned_addr learned my addr 172.21.15.44:0/33594174 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:43.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.191+0000 7f1a71ffb640 10 client.?.objecter ms_handle_connect 0x7f1a7c0a98f0 2024-03-25T17:43:43.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.191+0000 7f1a71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:43.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.201+0000 7f1a71ffb640 10 client.?.objecter ms_handle_connect 0x7f1a7c13f710 2024-03-25T17:43:43.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.201+0000 7f1a71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:43.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.203+0000 7f1a71ffb640 10 client.?.objecter ms_handle_connect 0x7f1a7c1430c0 2024-03-25T17:43:43.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.203+0000 7f1a71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:43.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.212+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034496053 0 0) 0x7f1a7c1370b0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.212+0000 7f1a71ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:43:43.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.212+0000 7f1a71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:43.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.212+0000 7f1a71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:43.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.212+0000 7f1a71ffb640 10 monclient(hunting): my global_id is 4427 2024-03-25T17:43:43.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.212+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a60004eb0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.214+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095239305 0 0) 0x7f1a7c136e60 con 0x7f1a7c1430c0 2024-03-25T17:43:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.214+0000 7f1a71ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:43:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.214+0000 7f1a71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.214+0000 7f1a71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.214+0000 7f1a71ffb640 10 monclient(hunting): my global_id is 4440 2024-03-25T17:43:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.214+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c1370b0 con 0x7f1a7c1430c0 2024-03-25T17:43:43.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.216+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835225223 0 0) 0x7f1a7c1381a0 con 0x7f1a7c13f710 2024-03-25T17:43:43.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.217+0000 7f1a71ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:43:43.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.217+0000 7f1a71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:43.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.217+0000 7f1a71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:43.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.217+0000 7f1a71ffb640 10 monclient(hunting): my global_id is 4429 2024-03-25T17:43:43.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.217+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c136e60 con 0x7f1a7c13f710 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.223+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762306353 0 0) 0x7f1a60004eb0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.223+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a7c1381a0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.223+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1a78001670 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.223+0000 7f1a71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.223+0000 7f1a71ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.223+0000 7f1a71ffb640 10 monclient(hunting): dump: 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:43.224 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:43.225 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:43.225 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:43.225 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:43.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.223+0000 7f1a71ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:43:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4241154910 0 0) 0x7f1a7c1381a0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 >> v1:172.21.15.111:6789/0 conn(0x7f1a7c13f710 legacy=0x7f1a7c13a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 >> v1:172.21.15.44:6789/0 conn(0x7f1a7c1430c0 legacy=0x7f1a7c13ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:43:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 1 monclient: found mon.c 2024-03-25T17:43:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c137390 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 10 monclient: _finish_auth 0 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a8390e640 5 monclient: authenticate success, global_id 4427 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a8390e640 10 monclient: _renew_subs 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a8390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c1376d0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a8390e640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a8390e640 10 monclient: _renew_subs 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a8390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.237+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c135530 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.238+0000 7f1a8390e640 1 librados: init done 2024-03-25T17:43:43.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.238+0000 7f1a8390e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:43.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.238+0000 7f1a8390e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:43.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.239+0000 7f1a8390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:43.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.239+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7c0b3ef0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.549+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a780038a0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.549+0000 7f1a71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.549+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1a78003cf0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.549+0000 7f1a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.549+0000 7f1a71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.549+0000 7f1a71ffb640 10 monclient: dump: 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:43.550 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:43.551 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:43.551 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:43.551 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:43.551 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:43.551 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:43.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.549+0000 7f1a737fe640 4 set_mon_vals no callback set 2024-03-25T17:43:43.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.550+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f1a78003f10 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.550+0000 7f1a80ea9640 10 monclient: get_auth_request con 0x7f1a6006ddf0 auth_method 0 2024-03-25T17:43:43.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.550+0000 7f1a71ffb640 10 client.4427.objecter ms_handle_connect 0x7f1a6006ddf0 2024-03-25T17:43:43.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.555+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f1a780549b0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.555+0000 7f1a71ffb640 10 client.4427.objecter ms_dispatch 0x7f1a7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:43:43.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.555+0000 7f1a71ffb640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:43:43.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.555+0000 7f1a71ffb640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:43:43.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.556+0000 7f1a71ffb640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-25T17:43:43.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.556+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a780504c0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.556+0000 7f1a71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:43.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.556+0000 7f1a71ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:43:43.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.803+0000 7f1a8390e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:43.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.803+0000 7f1a8390e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:43.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.803+0000 7f1a8390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:43.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.803+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a7c137520 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.833+0000 7f1a71ffb640 1 -- 172.21.15.44:0/33594174 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f1a780523f0 con 0x7f1a7c0a98f0 2024-03-25T17:43:43.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.833+0000 7f1a71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:43.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.833+0000 7f1a71ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:43:43.834 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:43:43.834 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 10 librados: watch_flush enter 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 10 librados: watch_flush exit 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 10 client.4427.objecter successfully canceled tick 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 >> v1:172.21.15.44:6800/3257043132 conn(0x7f1a6006ddf0 legacy=0x7f1a600702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 10 monclient: shutdown 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 >> v1:172.21.15.44:6790/0 conn(0x7f1a7c0a98f0 legacy=0x7f1a7c137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 shutdown_connections 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 >> 172.21.15.44:0/33594174 conn(0x7f1a7c0b7ea0 msgr2=0x7f1a7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:43.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.836+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 shutdown_connections 2024-03-25T17:43:43.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.837+0000 7f1a8390e640 1 -- 172.21.15.44:0/33594174 wait complete. 2024-03-25T17:43:43.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:43.837+0000 7f1a8390e640 1 librados: shutdown 2024-03-25T17:43:43.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:43:50.658 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:43:50.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.777+0000 7f772d616640 10 monclient: get_monmap_and_config 2024-03-25T17:43:50.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.777+0000 7f772d616640 10 monclient: build_initial_monmap 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.777+0000 7f772d616640 10 monclient: monmap: 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:43:50.778386+0000 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:43:50.778386+0000 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:43:50.778 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:50.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.778+0000 7f772d616640 1 Processor -- start 2024-03-25T17:43:50.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.778+0000 7f772d616640 1 -- start start 2024-03-25T17:43:50.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.778+0000 7f772d616640 10 monclient: init 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.778+0000 7f772d616640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.778+0000 7f772d616640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.778+0000 7f772d616640 10 monclient(hunting): picked mon.noname-a con 0x7f772813f990 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.778+0000 7f772d616640 10 monclient(hunting): picked mon.noname-b con 0x7f772813fd70 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.779+0000 7f772d616640 10 monclient(hunting): picked mon.noname-c con 0x7f77280afa60 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.779+0000 7f772d616640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280afeb0 con 0x7f772813f990 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.779+0000 7f772d616640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280b0100 con 0x7f77280afa60 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.779+0000 7f772d616640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280b1150 con 0x7f772813fd70 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.779+0000 7f772d616640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:50.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.779+0000 7f772d616640 10 monclient(hunting): authenticate will time out at 6234.006722s 2024-03-25T17:43:50.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.794+0000 7f77267fc640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f772813fd70 0x7f7728145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:37672/0 (socket says 172.21.15.44:37672) 2024-03-25T17:43:50.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.794+0000 7f77267fc640 1 -- 172.21.15.44:0/3550862318 learned_addr learned my addr 172.21.15.44:0/3550862318 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:50.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.817+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767089687 0 0) 0x7f77280b0100 con 0x7f77280afa60 2024-03-25T17:43:50.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.817+0000 7f7725ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:43:50.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.817+0000 7f7725ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:50.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.817+0000 7f7725ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:50.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.817+0000 7f7725ffb640 10 monclient(hunting): my global_id is 4432 2024-03-25T17:43:50.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.817+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7710003520 con 0x7f77280afa60 2024-03-25T17:43:50.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.823+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441555420 0 0) 0x7f77280b1150 con 0x7f772813fd70 2024-03-25T17:43:50.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.823+0000 7f7725ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:43:50.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.823+0000 7f7725ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:50.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.823+0000 7f7725ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:50.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.823+0000 7f7725ffb640 10 monclient(hunting): my global_id is 4430 2024-03-25T17:43:50.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.823+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77280b0100 con 0x7f772813fd70 2024-03-25T17:43:50.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2557403183 0 0) 0x7f7710003520 con 0x7f77280afa60 2024-03-25T17:43:50.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 >> v1:172.21.15.44:6790/0 conn(0x7f772813fd70 legacy=0x7f7728145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:50.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 >> v1:172.21.15.44:6789/0 conn(0x7f772813f990 legacy=0x7f772806ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:50.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:43:50.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 1 monclient: found mon.noname-c 2024-03-25T17:43:50.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:50.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280aede0 con 0x7f77280afa60 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 10 monclient: _finish_auth 0 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7720001a10 con 0x7f77280afa60 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f7725ffb640 10 monclient: dump: 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:50.827 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:50.828 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:50.828 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:50.828 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:50.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f772d616640 5 monclient: authenticate success, global_id 4432 2024-03-25T17:43:50.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.825+0000 7f772d616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:50.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7720001b70 con 0x7f77280afa60 2024-03-25T17:43:50.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f7725ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:50.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f7725ffb640 1 -- 172.21.15.44:0/3550862318 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7720001f80 con 0x7f77280afa60 2024-03-25T17:43:50.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f772d616640 10 monclient: get_monmap_and_config success 2024-03-25T17:43:50.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f772d616640 4 set_mon_vals no callback set 2024-03-25T17:43:50.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f7725ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f7725ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f7725ffb640 10 monclient: dump: 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:50.840 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:50.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f772d616640 10 monclient: shutdown 2024-03-25T17:43:50.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f772d616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:50.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.839+0000 7f772d616640 1 -- 172.21.15.44:0/3550862318 >> v1:172.21.15.111:6789/0 conn(0x7f77280afa60 legacy=0x7f772814dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.840+0000 7f772d616640 1 -- 172.21.15.44:0/3550862318 shutdown_connections 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.840+0000 7f772d616640 1 -- 172.21.15.44:0/3550862318 >> 172.21.15.44:0/3550862318 conn(0x7f77280b4b80 msgr2=0x7f77280b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.840+0000 7f772d616640 1 -- 172.21.15.44:0/3550862318 shutdown_connections 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.840+0000 7f772d616640 1 -- 172.21.15.44:0/3550862318 wait complete. 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.841+0000 7f772d616640 10 monclient: build_initial_monmap 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.841+0000 7f772d616640 10 monclient: monmap: 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:43:50.842 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:43:50.843 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:50.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.841+0000 7f772d616640 1 librados: starting msgr at 2024-03-25T17:43:50.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.841+0000 7f772d616640 1 librados: starting objecter 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.841+0000 7f772d616640 1 Processor -- start 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.842+0000 7f772d616640 1 -- start start 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.842+0000 7f772d616640 1 librados: setting wanted keys 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.842+0000 7f772d616640 1 librados: calling monclient init 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.842+0000 7f772d616640 10 monclient: init 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.842+0000 7f772d616640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.842+0000 7f772d616640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.842+0000 7f772d616640 10 monclient(hunting): picked mon.noname-a con 0x7f77280afa60 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.842+0000 7f772d616640 10 monclient(hunting): picked mon.noname-c con 0x7f772813f990 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.843+0000 7f772d616640 10 monclient(hunting): picked mon.noname-b con 0x7f772813fd70 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.843+0000 7f772d616640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77281d73f0 con 0x7f77280afa60 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.843+0000 7f772d616640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772813c910 con 0x7f772813fd70 2024-03-25T17:43:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.843+0000 7f772d616640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772811cc90 con 0x7f772813f990 2024-03-25T17:43:50.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.843+0000 7f772d616640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:50.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.843+0000 7f772d616640 10 monclient(hunting): authenticate will time out at 6234.070830s 2024-03-25T17:43:50.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.850+0000 7f77267fc640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f772813f990 0x7f772813be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:37736/0 (socket says 172.21.15.44:37736) 2024-03-25T17:43:50.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.850+0000 7f77267fc640 1 -- 172.21.15.44:0/1231271093 learned_addr learned my addr 172.21.15.44:0/1231271093 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:50.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.854+0000 7f770f7fe640 10 client.?.objecter ms_handle_connect 0x7f772813f990 2024-03-25T17:43:50.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.854+0000 7f770f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:50.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.862+0000 7f770f7fe640 10 client.?.objecter ms_handle_connect 0x7f772813fd70 2024-03-25T17:43:50.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.862+0000 7f770f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:50.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.866+0000 7f770f7fe640 10 client.?.objecter ms_handle_connect 0x7f77280afa60 2024-03-25T17:43:50.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.866+0000 7f770f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:50.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.868+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040273239 0 0) 0x7f772811cc90 con 0x7f772813f990 2024-03-25T17:43:50.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.868+0000 7f770f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:43:50.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.868+0000 7f770f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:50.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.868+0000 7f770f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:50.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.868+0000 7f770f7fe640 10 monclient(hunting): my global_id is 4433 2024-03-25T17:43:50.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.868+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7714004eb0 con 0x7f772813f990 2024-03-25T17:43:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.872+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010056137 0 0) 0x7f772813c910 con 0x7f772813fd70 2024-03-25T17:43:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.872+0000 7f770f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:43:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.872+0000 7f770f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.872+0000 7f770f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.872+0000 7f770f7fe640 10 monclient(hunting): my global_id is 4435 2024-03-25T17:43:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.872+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f772811cc90 con 0x7f772813fd70 2024-03-25T17:43:50.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.883+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238033438 0 0) 0x7f77281d73f0 con 0x7f77280afa60 2024-03-25T17:43:50.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.883+0000 7f770f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:43:50.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.883+0000 7f770f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:50.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.883+0000 7f770f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:50.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.883+0000 7f770f7fe640 10 monclient(hunting): my global_id is 4446 2024-03-25T17:43:50.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.883+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f772813c910 con 0x7f77280afa60 2024-03-25T17:43:50.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.890+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3366933182 0 0) 0x7f7714004eb0 con 0x7f772813f990 2024-03-25T17:43:50.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.890+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77281d73f0 con 0x7f772813f990 2024-03-25T17:43:50.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.890+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f771c001910 con 0x7f772813f990 2024-03-25T17:43:50.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.890+0000 7f770f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:50.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.890+0000 7f770f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:43:50.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.890+0000 7f770f7fe640 10 monclient(hunting): dump: 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:50.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.890+0000 7f770f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:43:50.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.898+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2002544648 0 0) 0x7f772811cc90 con 0x7f772813fd70 2024-03-25T17:43:50.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.898+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7714004eb0 con 0x7f772813fd70 2024-03-25T17:43:50.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.898+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7720001e80 con 0x7f772813fd70 2024-03-25T17:43:50.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.898+0000 7f770f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:50.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.898+0000 7f770f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:50.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.898+0000 7f770f7fe640 10 monclient(hunting): dump: 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:50.899 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.901+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1511690982 0 0) 0x7f772813c910 con 0x7f77280afa60 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.901+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f772811cc90 con 0x7f77280afa60 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.901+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7718001860 con 0x7f77280afa60 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.901+0000 7f770f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.901+0000 7f770f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.901+0000 7f770f7fe640 10 monclient(hunting): dump: 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:50.902 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:50.903 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:50.903 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 839516365 0 0) 0x7f77281d73f0 con 0x7f772813f990 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 >> v1:172.21.15.111:6789/0 conn(0x7f772813fd70 legacy=0x7f77281d78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 >> v1:172.21.15.44:6789/0 conn(0x7f77280afa60 legacy=0x7f772811da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 1 monclient: found mon.c 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772811d3e0 con 0x7f772813f990 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f770f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:50.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 5 monclient: authenticate success, global_id 4433 2024-03-25T17:43:50.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 10 monclient: _renew_subs 2024-03-25T17:43:50.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:50.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772811e170 con 0x7f772813f990 2024-03-25T17:43:50.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:43:50.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 10 monclient: _renew_subs 2024-03-25T17:43:50.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:50.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f772813c530 con 0x7f772813f990 2024-03-25T17:43:50.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.911+0000 7f772d616640 1 librados: init done 2024-03-25T17:43:50.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.913+0000 7f772d616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:50.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.913+0000 7f772d616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:50.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.913+0000 7f772d616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:50.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:50.913+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772806bac0 con 0x7f772813f990 2024-03-25T17:43:51.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.243+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f771c0047e0 con 0x7f772813f990 2024-03-25T17:43:51.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.243+0000 7f770f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:51.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.243+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f771c004af0 con 0x7f772813f990 2024-03-25T17:43:51.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.243+0000 7f770f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:51.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.243+0000 7f770f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.243+0000 7f770f7fe640 10 monclient: dump: 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:51.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.243+0000 7f77257fa640 4 set_mon_vals no callback set 2024-03-25T17:43:51.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.244+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f771c01aac0 con 0x7f772813f990 2024-03-25T17:43:51.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.245+0000 7f7726ffd640 10 monclient: get_auth_request con 0x7f771406e0f0 auth_method 0 2024-03-25T17:43:51.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.245+0000 7f770f7fe640 10 client.4433.objecter ms_handle_connect 0x7f771406e0f0 2024-03-25T17:43:51.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.251+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f771c0547a0 con 0x7f772813f990 2024-03-25T17:43:51.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.251+0000 7f770f7fe640 10 client.4433.objecter ms_dispatch 0x7f7728000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:43:51.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.251+0000 7f770f7fe640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:43:51.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.251+0000 7f770f7fe640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:43:51.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.251+0000 7f770f7fe640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-25T17:43:51.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.251+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f771c0500e0 con 0x7f772813f990 2024-03-25T17:43:51.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.251+0000 7f770f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:51.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.251+0000 7f770f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:43:51.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.494+0000 7f772d616640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:51.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.494+0000 7f772d616640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:51.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.494+0000 7f772d616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:43:51.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.494+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f772811f0f0 con 0x7f772813f990 2024-03-25T17:43:51.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.525+0000 7f770f7fe640 1 -- 172.21.15.44:0/1231271093 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f771c0521e0 con 0x7f772813f990 2024-03-25T17:43:51.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.525+0000 7f770f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:51.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.525+0000 7f770f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:43:51.526 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:43:51.526 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.527+0000 7f772d616640 10 librados: watch_flush enter 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.527+0000 7f772d616640 10 librados: watch_flush exit 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.527+0000 7f772d616640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.527+0000 7f772d616640 10 client.4433.objecter successfully canceled tick 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 >> v1:172.21.15.44:6800/3257043132 conn(0x7f771406e0f0 legacy=0x7f77140705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 10 monclient: shutdown 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 >> v1:172.21.15.44:6790/0 conn(0x7f772813f990 legacy=0x7f772813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 shutdown_connections 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 >> 172.21.15.44:0/1231271093 conn(0x7f77280b4b80 msgr2=0x7f772806f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 shutdown_connections 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 1 -- 172.21.15.44:0/1231271093 wait complete. 2024-03-25T17:43:51.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:51.528+0000 7f772d616640 1 librados: shutdown 2024-03-25T17:43:51.541 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:43:57.896 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:15.252 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:57.954 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:15.310 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:57.979 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:15.335 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:58.005 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:15.325 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:58.008 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:15.328 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:58.012 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:15.368 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:58.019 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:15.375 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:58.034 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:15.354 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:58.049 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:15.405 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:43:58.347 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:43:58.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.463+0000 7fdb48772640 10 monclient: get_monmap_and_config 2024-03-25T17:43:58.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.463+0000 7fdb48772640 10 monclient: build_initial_monmap 2024-03-25T17:43:58.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.463+0000 7fdb48772640 10 monclient: monmap: 2024-03-25T17:43:58.463 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:43:58.464211+0000 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:43:58.464211+0000 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:43:58.464 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:58.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.463+0000 7fdb48772640 1 Processor -- start 2024-03-25T17:43:58.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.463+0000 7fdb48772640 1 -- start start 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.463+0000 7fdb48772640 10 monclient: init 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 10 monclient(hunting): picked mon.noname-b con 0x7fdb400b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 10 monclient(hunting): picked mon.noname-a con 0x7fdb400b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 10 monclient(hunting): picked mon.noname-c con 0x7fdb4014be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400af940 con 0x7fdb400b1730 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400b0a70 con 0x7fdb4014be90 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400b0d50 con 0x7fdb400b4d60 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:58.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.464+0000 7fdb48772640 10 monclient(hunting): authenticate will time out at 6241.692539s 2024-03-25T17:43:58.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.470+0000 7fdb4650e640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fdb400b4d60 0x7fdb4013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:42266/0 (socket says 172.21.15.44:42266) 2024-03-25T17:43:58.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.470+0000 7fdb4650e640 1 -- 172.21.15.44:0/2038483296 learned_addr learned my addr 172.21.15.44:0/2038483296 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:58.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.487+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933640612 0 0) 0x7fdb400b0a70 con 0x7fdb4014be90 2024-03-25T17:43:58.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.487+0000 7fdb4550c640 10 monclient(hunting): _negotiate 2024-03-25T17:43:58.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.487+0000 7fdb4550c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:58.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.487+0000 7fdb4550c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:58.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.487+0000 7fdb4550c640 10 monclient(hunting): my global_id is 4438 2024-03-25T17:43:58.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.487+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb2c003520 con 0x7fdb4014be90 2024-03-25T17:43:58.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.493+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886784490 0 0) 0x7fdb400b0d50 con 0x7fdb400b4d60 2024-03-25T17:43:58.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.493+0000 7fdb4550c640 10 monclient(hunting): _negotiate 2024-03-25T17:43:58.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.493+0000 7fdb4550c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:58.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.493+0000 7fdb4550c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:58.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.493+0000 7fdb4550c640 10 monclient(hunting): my global_id is 4436 2024-03-25T17:43:58.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.493+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb400b0a70 con 0x7fdb400b4d60 2024-03-25T17:43:58.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.494+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707656382 0 0) 0x7fdb400af940 con 0x7fdb400b1730 2024-03-25T17:43:58.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.494+0000 7fdb4550c640 10 monclient(hunting): _negotiate 2024-03-25T17:43:58.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.494+0000 7fdb4550c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:58.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.494+0000 7fdb4550c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:58.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.494+0000 7fdb4550c640 10 monclient(hunting): my global_id is 4449 2024-03-25T17:43:58.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.494+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb400b0d50 con 0x7fdb400b1730 2024-03-25T17:43:58.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.498+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 824347106 0 0) 0x7fdb2c003520 con 0x7fdb4014be90 2024-03-25T17:43:58.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.498+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 >> v1:172.21.15.44:6790/0 conn(0x7fdb400b4d60 legacy=0x7fdb4013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:58.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.498+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 >> v1:172.21.15.44:6789/0 conn(0x7fdb400b1730 legacy=0x7fdb400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:58.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 10 monclient: _finish_hunting 0 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 1 monclient: found mon.noname-c 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400b1030 con 0x7fdb4014be90 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 10 monclient: _finish_auth 0 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 10 monclient: _check_auth_tickets 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdb38001a10 con 0x7fdb4014be90 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb48772640 5 monclient: authenticate success, global_id 4438 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb48772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb4550c640 10 monclient: dump: 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.500 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.501 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:58.501 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:58.501 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:58.501 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:58.501 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:58.501 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:58.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.499+0000 7fdb48772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:43:58.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb38001b70 con 0x7fdb4014be90 2024-03-25T17:43:58.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb4550c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb4550c640 1 -- 172.21.15.44:0/2038483296 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdb38001f80 con 0x7fdb4014be90 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb48772640 10 monclient: get_monmap_and_config success 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb48772640 4 set_mon_vals no callback set 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb4550c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb4550c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb4550c640 10 monclient: dump: 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:58.523 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:58.524 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:58.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb48772640 10 monclient: shutdown 2024-03-25T17:43:58.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb48772640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:58.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.522+0000 7fdb48772640 1 -- 172.21.15.44:0/2038483296 >> v1:172.21.15.111:6789/0 conn(0x7fdb4014be90 legacy=0x7fdb4014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.523+0000 7fdb48772640 1 -- 172.21.15.44:0/2038483296 shutdown_connections 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.523+0000 7fdb48772640 1 -- 172.21.15.44:0/2038483296 >> 172.21.15.44:0/2038483296 conn(0x7fdb400617a0 msgr2=0x7fdb40061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.523+0000 7fdb48772640 1 -- 172.21.15.44:0/2038483296 shutdown_connections 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.523+0000 7fdb48772640 1 -- 172.21.15.44:0/2038483296 wait complete. 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.523+0000 7fdb48772640 10 monclient: build_initial_monmap 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.523+0000 7fdb48772640 10 monclient: monmap: 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:43:58.526 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 1 librados: starting msgr at 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 1 librados: starting objecter 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 1 Processor -- start 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 1 -- start start 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 1 librados: setting wanted keys 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 1 librados: calling monclient init 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 10 monclient: init 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 10 monclient: _reopen_session rank -1 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 10 monclient(hunting): picked mon.noname-c con 0x7fdb400b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 10 monclient(hunting): picked mon.noname-a con 0x7fdb400b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 10 monclient(hunting): picked mon.noname-b con 0x7fdb4014be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:43:58.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.524+0000 7fdb48772640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb401d78a0 con 0x7fdb400b4d60 2024-03-25T17:43:58.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.525+0000 7fdb48772640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb40139630 con 0x7fdb4014be90 2024-03-25T17:43:58.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.525+0000 7fdb48772640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb401d8520 con 0x7fdb400b1730 2024-03-25T17:43:58.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.525+0000 7fdb48772640 10 monclient(hunting): _renew_subs 2024-03-25T17:43:58.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.525+0000 7fdb48772640 10 monclient(hunting): authenticate will time out at 6241.752718s 2024-03-25T17:43:58.533 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.533+0000 7fdb45d0d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdb400b4d60 0x7fdb4013b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44324/0 (socket says 172.21.15.44:44324) 2024-03-25T17:43:58.533 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.533+0000 7fdb45d0d640 1 -- 172.21.15.44:0/2474035693 learned_addr learned my addr 172.21.15.44:0/2474035693 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:43:58.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.534+0000 7fdb36ffd640 10 client.?.objecter ms_handle_connect 0x7fdb400b4d60 2024-03-25T17:43:58.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.534+0000 7fdb36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:58.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.543+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126432328 0 0) 0x7fdb401d78a0 con 0x7fdb400b4d60 2024-03-25T17:43:58.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.543+0000 7fdb36ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:58.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.543+0000 7fdb36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:58.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.543+0000 7fdb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:58.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.543+0000 7fdb36ffd640 10 monclient(hunting): my global_id is 4452 2024-03-25T17:43:58.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.544+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb24004eb0 con 0x7fdb400b4d60 2024-03-25T17:43:58.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.544+0000 7fdb36ffd640 10 client.?.objecter ms_handle_connect 0x7fdb400b1730 2024-03-25T17:43:58.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.544+0000 7fdb36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:58.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.549+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098888063 0 0) 0x7fdb401d8520 con 0x7fdb400b1730 2024-03-25T17:43:58.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.549+0000 7fdb36ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:43:58.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.549+0000 7fdb36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:43:58.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.549+0000 7fdb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:43:58.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.549+0000 7fdb36ffd640 10 monclient(hunting): my global_id is 4439 2024-03-25T17:43:58.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.549+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb401d78a0 con 0x7fdb400b1730 2024-03-25T17:43:58.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.549+0000 7fdb36ffd640 10 client.?.objecter ms_handle_connect 0x7fdb4014be90 2024-03-25T17:43:58.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.549+0000 7fdb36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:43:58.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.553+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942306764 0 0) 0x7fdb24004eb0 con 0x7fdb400b4d60 2024-03-25T17:43:58.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.553+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb401d8520 con 0x7fdb400b4d60 2024-03-25T17:43:58.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.553+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdb30001a10 con 0x7fdb400b4d60 2024-03-25T17:43:58.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.553+0000 7fdb36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:58.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.553+0000 7fdb36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:43:58.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.553+0000 7fdb36ffd640 10 monclient(hunting): dump: 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:58.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.553+0000 7fdb36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.562+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3041644379 0 0) 0x7fdb401d78a0 con 0x7fdb400b1730 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.562+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb24004eb0 con 0x7fdb400b1730 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.562+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdb3c001970 con 0x7fdb400b1730 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.562+0000 7fdb36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.562+0000 7fdb36ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.562+0000 7fdb36ffd640 10 monclient(hunting): dump: 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.563 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.564 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:58.564 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:58.564 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:58.564 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:58.564 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:58.564 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:58.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2190938659 0 0) 0x7fdb401d8520 con 0x7fdb400b4d60 2024-03-25T17:43:58.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 >> v1:172.21.15.44:6790/0 conn(0x7fdb400b1730 legacy=0x7fdb401d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 >> v1:172.21.15.111:6789/0 conn(0x7fdb4014be90 legacy=0x7fdb401e30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:43:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 1 monclient: found mon.a 2024-03-25T17:43:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:43:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb401d86b0 con 0x7fdb400b4d60 2024-03-25T17:43:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 10 monclient: _finish_auth 0 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 5 monclient: authenticate success, global_id 4452 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 10 monclient: _renew_subs 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb40145570 con 0x7fdb400b4d60 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 10 monclient: _renew_subs 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4013e2b0 con 0x7fdb400b4d60 2024-03-25T17:43:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.567+0000 7fdb48772640 1 librados: init done 2024-03-25T17:43:58.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.568+0000 7fdb48772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:58.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.568+0000 7fdb48772640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:58.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.568+0000 7fdb48772640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:58.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.568+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb401397c0 con 0x7fdb400b4d60 2024-03-25T17:43:58.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.592+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb30001c70 con 0x7fdb400b4d60 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.592+0000 7fdb36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.592+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdb30003a60 con 0x7fdb400b4d60 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.592+0000 7fdb36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.592+0000 7fdb36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.592+0000 7fdb36ffd640 10 monclient: dump: 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:43:58.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.592+0000 7fdb44d0b640 4 set_mon_vals no callback set 2024-03-25T17:43:58.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.593+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fdb30003c80 con 0x7fdb400b4d60 2024-03-25T17:43:58.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.594+0000 7fdb4650e640 10 monclient: get_auth_request con 0x7fdb2406df20 auth_method 0 2024-03-25T17:43:58.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.594+0000 7fdb36ffd640 10 client.4452.objecter ms_handle_connect 0x7fdb2406df20 2024-03-25T17:43:58.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.596+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fdb30054810 con 0x7fdb400b4d60 2024-03-25T17:43:58.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.596+0000 7fdb36ffd640 10 client.4452.objecter ms_dispatch 0x7fdb40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:43:58.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.596+0000 7fdb36ffd640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:43:58.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.596+0000 7fdb36ffd640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:43:58.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.596+0000 7fdb36ffd640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-25T17:43:58.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.888+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb30050320 con 0x7fdb400b4d60 2024-03-25T17:43:58.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.888+0000 7fdb36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:43:58.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:58.888+0000 7fdb36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:43:59.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.109+0000 7fdb48772640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:59.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.109+0000 7fdb48772640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:59.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.109+0000 7fdb48772640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:43:59.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.109+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb400627b0 con 0x7fdb400b4d60 2024-03-25T17:43:59.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.144+0000 7fdb36ffd640 1 -- 172.21.15.44:0/2474035693 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fdb30052250 con 0x7fdb400b4d60 2024-03-25T17:43:59.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.144+0000 7fdb36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:43:59.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.144+0000 7fdb36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:43:59.145 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:43:59.145 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 10 librados: watch_flush enter 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 10 librados: watch_flush exit 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 10 client.4452.objecter successfully canceled tick 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 >> v1:172.21.15.44:6800/3257043132 conn(0x7fdb2406df20 legacy=0x7fdb240703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 10 monclient: shutdown 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 >> v1:172.21.15.44:6789/0 conn(0x7fdb400b4d60 legacy=0x7fdb4013b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 shutdown_connections 2024-03-25T17:43:59.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.146+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 >> 172.21.15.44:0/2474035693 conn(0x7fdb400617a0 msgr2=0x7fdb400b6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:43:59.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.147+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 shutdown_connections 2024-03-25T17:43:59.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.147+0000 7fdb48772640 1 -- 172.21.15.44:0/2474035693 wait complete. 2024-03-25T17:43:59.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:43:59.147+0000 7fdb48772640 1 librados: shutdown 2024-03-25T17:43:59.157 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:44:05.964 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:44:06.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.079+0000 7f608417b640 10 monclient: get_monmap_and_config 2024-03-25T17:44:06.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.079+0000 7f608417b640 10 monclient: build_initial_monmap 2024-03-25T17:44:06.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.079+0000 7f608417b640 10 monclient: monmap: 2024-03-25T17:44:06.081 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:06.081 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:06.081 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:44:06.081798+0000 2024-03-25T17:44:06.081 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:44:06.081798+0000 2024-03-25T17:44:06.081 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:06.082 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:06.082 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:06.082 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:44:06.082 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:44:06.082 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:06.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.080+0000 7f608417b640 1 Processor -- start 2024-03-25T17:44:06.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.081+0000 7f608417b640 1 -- start start 2024-03-25T17:44:06.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.081+0000 7f608417b640 10 monclient: init 2024-03-25T17:44:06.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.081+0000 7f608417b640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:06.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.081+0000 7f608417b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:44:06.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.081+0000 7f608417b640 10 monclient(hunting): picked mon.noname-b con 0x7f607c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:06.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.082+0000 7f608417b640 10 monclient(hunting): picked mon.noname-a con 0x7f607c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:06.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.082+0000 7f608417b640 10 monclient(hunting): picked mon.noname-c con 0x7f607c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:06.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.082+0000 7f608417b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c142520 con 0x7f607c13f710 2024-03-25T17:44:06.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.082+0000 7f608417b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c0b28a0 con 0x7f607c1430c0 2024-03-25T17:44:06.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.082+0000 7f608417b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c0b2a30 con 0x7f607c0a98f0 2024-03-25T17:44:06.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.082+0000 7f608417b640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:06.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.082+0000 7f608417b640 10 monclient(hunting): authenticate will time out at 6249.310763s 2024-03-25T17:44:06.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.096+0000 7f6082718640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f607c1430c0 0x7f607c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:40660/0 (socket says 172.21.15.44:40660) 2024-03-25T17:44:06.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.096+0000 7f6082718640 1 -- 172.21.15.44:0/3092147292 learned_addr learned my addr 172.21.15.44:0/3092147292 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:06.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.099+0000 7f6082718640 1 -- 172.21.15.44:0/3092147292 >> v1:172.21.15.111:6789/0 conn(0x7f607c1430c0 legacy=0x7f607c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T17:44:06.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.099+0000 7f6082718640 1 -- 172.21.15.44:0/3092147292 >> v1:172.21.15.111:6789/0 conn(0x7f607c1430c0 legacy=0x7f607c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:44:06.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.099+0000 7f6082718640 1 --1- 172.21.15.44:0/3092147292 >> v1:172.21.15.111:6789/0 conn(0x7f607c1430c0 0x7f607c14dc30 :-1 s=OPENED pgs=35073 cs=1 l=1).handle_message read tag failed 2024-03-25T17:44:06.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.099+0000 7f6082718640 1 --1- 172.21.15.44:0/3092147292 >> v1:172.21.15.111:6789/0 conn(0x7f607c1430c0 0x7f607c14dc30 :-1 s=OPENED pgs=35073 cs=1 l=1).fault on lossy channel, failing 2024-03-25T17:44:06.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.099+0000 7f6080f15640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.111:6789/0 2024-03-25T17:44:06.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817551335 0 0) 0x7f607c142520 con 0x7f607c13f710 2024-03-25T17:44:06.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 10 monclient(hunting): _negotiate 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 10 monclient(hunting): my global_id is 4455 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6068003520 con 0x7f607c13f710 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075015036 0 0) 0x7f607c0b2a30 con 0x7f607c0a98f0 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 10 monclient(hunting): _negotiate 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 10 monclient(hunting): my global_id is 4442 2024-03-25T17:44:06.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.114+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607c142520 con 0x7f607c0a98f0 2024-03-25T17:44:06.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674304578 0 0) 0x7f6068003520 con 0x7f607c13f710 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 >> v1:172.21.15.44:6790/0 conn(0x7f607c0a98f0 legacy=0x7f607c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 >> v1:172.21.15.111:6789/0 conn(0x7f607c1430c0 legacy=0x7f607c14dc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 10 monclient: _finish_hunting 0 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 1 monclient: found mon.noname-a 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c0b44c0 con 0x7f607c13f710 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 10 monclient: _finish_auth 0 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 10 monclient: _check_auth_tickets 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f606c001690 con 0x7f607c13f710 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f6080f15640 10 monclient: dump: 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:06.195 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f608417b640 5 monclient: authenticate success, global_id 4455 2024-03-25T17:44:06.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.193+0000 7f608417b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:06.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f606c0017f0 con 0x7f607c13f710 2024-03-25T17:44:06.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f6080f15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:06.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f6080f15640 1 -- 172.21.15.44:0/3092147292 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f606c001c00 con 0x7f607c13f710 2024-03-25T17:44:06.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f6080f15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:06.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f6080f15640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:06.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f6080f15640 10 monclient: dump: 2024-03-25T17:44:06.209 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:06.209 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f608417b640 10 monclient: get_monmap_and_config success 2024-03-25T17:44:06.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f608417b640 4 set_mon_vals no callback set 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f608417b640 10 monclient: shutdown 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f608417b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f608417b640 1 -- 172.21.15.44:0/3092147292 >> v1:172.21.15.44:6789/0 conn(0x7f607c13f710 legacy=0x7f607c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f608417b640 1 -- 172.21.15.44:0/3092147292 shutdown_connections 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.208+0000 7f608417b640 1 -- 172.21.15.44:0/3092147292 >> 172.21.15.44:0/3092147292 conn(0x7f607c0b7ea0 msgr2=0x7f607c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.209+0000 7f608417b640 1 -- 172.21.15.44:0/3092147292 shutdown_connections 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.209+0000 7f608417b640 1 -- 172.21.15.44:0/3092147292 wait complete. 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.209+0000 7f608417b640 10 monclient: build_initial_monmap 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.209+0000 7f608417b640 10 monclient: monmap: 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:44:06.213 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.210+0000 7f608417b640 1 librados: starting msgr at 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.210+0000 7f608417b640 1 librados: starting objecter 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.210+0000 7f608417b640 1 Processor -- start 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.210+0000 7f608417b640 1 -- start start 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.210+0000 7f608417b640 1 librados: setting wanted keys 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.210+0000 7f608417b640 1 librados: calling monclient init 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.210+0000 7f608417b640 10 monclient: init 2024-03-25T17:44:06.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 10 monclient(hunting): picked mon.noname-c con 0x7f607c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 10 monclient(hunting): picked mon.noname-a con 0x7f607c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 10 monclient(hunting): picked mon.noname-b con 0x7f607c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c1d78e0 con 0x7f607c13f710 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c0b2450 con 0x7f607c1430c0 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c1d7490 con 0x7f607c0a98f0 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:06.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.211+0000 7f608417b640 10 monclient(hunting): authenticate will time out at 6249.440404s 2024-03-25T17:44:06.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.223+0000 7f6081716640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f607c13f710 0x7f607c0af090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:36126/0 (socket says 172.21.15.44:36126) 2024-03-25T17:44:06.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.223+0000 7f6081716640 1 -- 172.21.15.44:0/3371378684 learned_addr learned my addr 172.21.15.44:0/3371378684 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:06.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.231+0000 7f60727fc640 10 client.?.objecter ms_handle_connect 0x7f607c13f710 2024-03-25T17:44:06.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.231+0000 7f60727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:06.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.247+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196221892 0 0) 0x7f607c1d78e0 con 0x7f607c13f710 2024-03-25T17:44:06.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.247+0000 7f60727fc640 10 monclient(hunting): _negotiate 2024-03-25T17:44:06.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.247+0000 7f60727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:06.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.247+0000 7f60727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:06.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.247+0000 7f60727fc640 10 monclient(hunting): my global_id is 4458 2024-03-25T17:44:06.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.247+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6060004eb0 con 0x7f607c13f710 2024-03-25T17:44:06.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.259+0000 7f60727fc640 10 client.?.objecter ms_handle_connect 0x7f607c1430c0 2024-03-25T17:44:06.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.259+0000 7f60727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:06.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.261+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 410675542 0 0) 0x7f6060004eb0 con 0x7f607c13f710 2024-03-25T17:44:06.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.261+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f607c1d78e0 con 0x7f607c13f710 2024-03-25T17:44:06.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.261+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f606c001bc0 con 0x7f607c13f710 2024-03-25T17:44:06.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.261+0000 7f60727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:06.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.261+0000 7f60727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.261+0000 7f60727fc640 10 monclient(hunting): dump: 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:06.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.261+0000 7f60727fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:44:06.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.268+0000 7f60727fc640 10 client.?.objecter ms_handle_connect 0x7f607c0a98f0 2024-03-25T17:44:06.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.268+0000 7f60727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:06.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.275+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895352959 0 0) 0x7f607c0b2450 con 0x7f607c1430c0 2024-03-25T17:44:06.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.275+0000 7f60727fc640 10 monclient(hunting): _negotiate 2024-03-25T17:44:06.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.275+0000 7f60727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:06.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.275+0000 7f60727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:06.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.275+0000 7f60727fc640 10 monclient(hunting): my global_id is 4444 2024-03-25T17:44:06.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.275+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6060004eb0 con 0x7f607c1430c0 2024-03-25T17:44:06.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.289+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058035224 0 0) 0x7f607c1d7490 con 0x7f607c0a98f0 2024-03-25T17:44:06.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.289+0000 7f60727fc640 10 monclient(hunting): _negotiate 2024-03-25T17:44:06.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.289+0000 7f60727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:06.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.289+0000 7f60727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:06.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.289+0000 7f60727fc640 10 monclient(hunting): my global_id is 4445 2024-03-25T17:44:06.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.289+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607c0b2450 con 0x7f607c0a98f0 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.291+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 111191929 0 0) 0x7f6060004eb0 con 0x7f607c1430c0 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.291+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f607c1d7490 con 0x7f607c1430c0 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.292+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6074001b40 con 0x7f607c1430c0 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.292+0000 7f60727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.292+0000 7f60727fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.292+0000 7f60727fc640 10 monclient(hunting): dump: 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:06.293 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.294 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.294 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:06.294 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:06.294 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:06.294 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:06.294 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:06.294 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.299+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042762083 0 0) 0x7f607c0b2450 con 0x7f607c0a98f0 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.299+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6060004eb0 con 0x7f607c0a98f0 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.299+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f60780018f0 con 0x7f607c0a98f0 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.299+0000 7f60727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.299+0000 7f60727fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.299+0000 7f60727fc640 10 monclient(hunting): dump: 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.301 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:06.302 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:06.302 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:06.302 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:06.302 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:06.302 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:06.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 997812375 0 0) 0x7f607c1d78e0 con 0x7f607c13f710 2024-03-25T17:44:06.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 >> v1:172.21.15.44:6790/0 conn(0x7f607c0a98f0 legacy=0x7f607c1d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:06.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 >> v1:172.21.15.111:6789/0 conn(0x7f607c1430c0 legacy=0x7f607c0afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:06.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 10 monclient: _finish_hunting 0 2024-03-25T17:44:06.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 1 monclient: found mon.a 2024-03-25T17:44:06.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c1d7620 con 0x7f607c13f710 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 10 monclient: _finish_auth 0 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 10 monclient: _check_auth_tickets 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f60727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 5 monclient: authenticate success, global_id 4458 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 10 monclient: _renew_subs 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607c14dd70 con 0x7f607c13f710 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 10 monclient: _renew_subs 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f607c0b1e80 con 0x7f607c13f710 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.300+0000 7f608417b640 1 librados: init done 2024-03-25T17:44:06.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.301+0000 7f608417b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:06.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.301+0000 7f608417b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:06.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.301+0000 7f608417b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:06.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.301+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607c074cc0 con 0x7f607c13f710 2024-03-25T17:44:06.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.623+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f606c0038a0 con 0x7f607c13f710 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.623+0000 7f60727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.623+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f606c003cb0 con 0x7f607c13f710 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.623+0000 7f60727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.623+0000 7f60727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.623+0000 7f60727fc640 10 monclient: dump: 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:06.625 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:06.626 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:06.626 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.623+0000 7f6073fff640 4 set_mon_vals no callback set 2024-03-25T17:44:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.624+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f606c003ed0 con 0x7f607c13f710 2024-03-25T17:44:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.624+0000 7f6081f17640 10 monclient: get_auth_request con 0x7f606006e180 auth_method 0 2024-03-25T17:44:06.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.625+0000 7f60727fc640 10 client.4458.objecter ms_handle_connect 0x7f606006e180 2024-03-25T17:44:06.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.632+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f606c0548f0 con 0x7f607c13f710 2024-03-25T17:44:06.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.632+0000 7f60727fc640 10 client.4458.objecter ms_dispatch 0x7f607c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:44:06.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.632+0000 7f60727fc640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:44:06.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.632+0000 7f60727fc640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:44:06.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.632+0000 7f60727fc640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-25T17:44:06.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.632+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f606c050b90 con 0x7f607c13f710 2024-03-25T17:44:06.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.632+0000 7f60727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:06.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.632+0000 7f60727fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:44:06.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.893+0000 7f608417b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:06.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.893+0000 7f608417b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:06.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.893+0000 7f608417b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:06.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.893+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f607c0627b0 con 0x7f607c13f710 2024-03-25T17:44:06.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.921+0000 7f60727fc640 1 -- 172.21.15.44:0/3371378684 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f606c050d30 con 0x7f607c13f710 2024-03-25T17:44:06.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.921+0000 7f60727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:06.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.921+0000 7f60727fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:44:06.923 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:44:06.923 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:44:06.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.923+0000 7f608417b640 10 librados: watch_flush enter 2024-03-25T17:44:06.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.923+0000 7f608417b640 10 librados: watch_flush exit 2024-03-25T17:44:06.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.923+0000 7f608417b640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-25T17:44:06.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.923+0000 7f608417b640 10 client.4458.objecter successfully canceled tick 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 >> v1:172.21.15.44:6800/3257043132 conn(0x7f606006e180 legacy=0x7f6060070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 10 monclient: shutdown 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 >> v1:172.21.15.44:6789/0 conn(0x7f607c13f710 legacy=0x7f607c0af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 shutdown_connections 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 >> 172.21.15.44:0/3371378684 conn(0x7f607c0b7ea0 msgr2=0x7f607c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 shutdown_connections 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 1 -- 172.21.15.44:0/3371378684 wait complete. 2024-03-25T17:44:06.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:06.924+0000 7f608417b640 1 librados: shutdown 2024-03-25T17:44:06.937 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:44:13.744 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.854+0000 7f221f298640 10 monclient: get_monmap_and_config 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.854+0000 7f221f298640 10 monclient: build_initial_monmap 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.854+0000 7f221f298640 10 monclient: monmap: 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:44:13.855675+0000 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:44:13.855675+0000 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:13.855 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:13.856 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:44:13.856 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:44:13.856 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.855+0000 7f221f298640 1 Processor -- start 2024-03-25T17:44:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.855+0000 7f221f298640 1 -- start start 2024-03-25T17:44:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.855+0000 7f221f298640 10 monclient: init 2024-03-25T17:44:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:44:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 10 monclient(hunting): picked mon.noname-a con 0x7f22180a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 10 monclient(hunting): picked mon.noname-c con 0x7f221813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 10 monclient(hunting): picked mon.noname-b con 0x7f22181430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218142520 con 0x7f22180a98f0 2024-03-25T17:44:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22180b28a0 con 0x7f221813f710 2024-03-25T17:44:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22180b2a30 con 0x7f22181430c0 2024-03-25T17:44:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.856+0000 7f221f298640 10 monclient(hunting): authenticate will time out at 6257.084442s 2024-03-25T17:44:13.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.864+0000 7f221d835640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f22181430c0 0x7f221814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:57070/0 (socket says 172.21.15.44:57070) 2024-03-25T17:44:13.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.864+0000 7f221d835640 1 -- 172.21.15.44:0/3418592907 learned_addr learned my addr 172.21.15.44:0/3418592907 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:13.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.873+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195923502 0 0) 0x7f22180b2a30 con 0x7f22181430c0 2024-03-25T17:44:13.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.873+0000 7f220ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:44:13.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.873+0000 7f220ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:13.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.873+0000 7f220ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:13.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.873+0000 7f220ffff640 10 monclient(hunting): my global_id is 4448 2024-03-25T17:44:13.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.873+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2204003520 con 0x7f22181430c0 2024-03-25T17:44:13.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.883+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655234024 0 0) 0x7f2218142520 con 0x7f22180a98f0 2024-03-25T17:44:13.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.883+0000 7f220ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:44:13.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.883+0000 7f220ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:13.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.883+0000 7f220ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:13.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.883+0000 7f220ffff640 10 monclient(hunting): my global_id is 4461 2024-03-25T17:44:13.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.883+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22180b2a30 con 0x7f22180a98f0 2024-03-25T17:44:13.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.886+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197324700 0 0) 0x7f22180b28a0 con 0x7f221813f710 2024-03-25T17:44:13.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.886+0000 7f220ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:44:13.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.886+0000 7f220ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:13.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.886+0000 7f220ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:13.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.886+0000 7f220ffff640 10 monclient(hunting): my global_id is 4447 2024-03-25T17:44:13.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.886+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2218142520 con 0x7f221813f710 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2574628919 0 0) 0x7f2204003520 con 0x7f22181430c0 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 >> v1:172.21.15.111:6789/0 conn(0x7f221813f710 legacy=0x7f22181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 >> v1:172.21.15.44:6789/0 conn(0x7f22180a98f0 legacy=0x7f22180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 1 monclient: found mon.noname-b 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22180b44c0 con 0x7f22181430c0 2024-03-25T17:44:13.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 10 monclient: _finish_auth 0 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2210001a10 con 0x7f22181430c0 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f221f298640 5 monclient: authenticate success, global_id 4448 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f221f298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f220ffff640 10 monclient: dump: 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:13.891 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:13.892 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:13.892 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:13.892 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:13.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.889+0000 7f221f298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.907+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2210001b70 con 0x7f22181430c0 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.907+0000 7f220ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.907+0000 7f220ffff640 1 -- 172.21.15.44:0/3418592907 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2210001f80 con 0x7f22181430c0 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.907+0000 7f221f298640 10 monclient: get_monmap_and_config success 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.907+0000 7f221f298640 4 set_mon_vals no callback set 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.907+0000 7f220ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.907+0000 7f220ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.907+0000 7f220ffff640 10 monclient: dump: 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:13.908 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:13.909 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.909 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.909 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:13.909 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:13.909 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:13.909 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:13.909 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:13.909 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:13.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.908+0000 7f221f298640 10 monclient: shutdown 2024-03-25T17:44:13.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.908+0000 7f221f298640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.908+0000 7f221f298640 1 -- 172.21.15.44:0/3418592907 >> v1:172.21.15.44:6790/0 conn(0x7f22181430c0 legacy=0x7f221814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.908+0000 7f221f298640 1 -- 172.21.15.44:0/3418592907 shutdown_connections 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.908+0000 7f221f298640 1 -- 172.21.15.44:0/3418592907 >> 172.21.15.44:0/3418592907 conn(0x7f22180b7ea0 msgr2=0x7f22180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.908+0000 7f221f298640 1 -- 172.21.15.44:0/3418592907 shutdown_connections 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.908+0000 7f221f298640 1 -- 172.21.15.44:0/3418592907 wait complete. 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.909+0000 7f221f298640 10 monclient: build_initial_monmap 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.909+0000 7f221f298640 10 monclient: monmap: 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:44:13.911 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.910+0000 7f221f298640 1 librados: starting msgr at 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.910+0000 7f221f298640 1 librados: starting objecter 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.910+0000 7f221f298640 1 Processor -- start 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.910+0000 7f221f298640 1 -- start start 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.910+0000 7f221f298640 1 librados: setting wanted keys 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.910+0000 7f221f298640 1 librados: calling monclient init 2024-03-25T17:44:13.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.910+0000 7f221f298640 10 monclient: init 2024-03-25T17:44:13.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:13.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:44:13.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 10 monclient(hunting): picked mon.noname-c con 0x7f22180a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:13.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 10 monclient(hunting): picked mon.noname-b con 0x7f221813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:13.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 10 monclient(hunting): picked mon.noname-a con 0x7f22181430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:13.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22181396e0 con 0x7f22181430c0 2024-03-25T17:44:13.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218138530 con 0x7f221813f710 2024-03-25T17:44:13.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22181399c0 con 0x7f22180a98f0 2024-03-25T17:44:13.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:13.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.911+0000 7f221f298640 10 monclient(hunting): authenticate will time out at 6257.139122s 2024-03-25T17:44:13.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.927+0000 7f221d835640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f22181430c0 0x7f22181e1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51550/0 (socket says 172.21.15.44:51550) 2024-03-25T17:44:13.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.927+0000 7f221d835640 1 -- 172.21.15.44:0/3560856471 learned_addr learned my addr 172.21.15.44:0/3560856471 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:13.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.927+0000 7f220dffb640 10 client.?.objecter ms_handle_connect 0x7f22181430c0 2024-03-25T17:44:13.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.928+0000 7f220dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:13.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.929+0000 7f220dffb640 10 client.?.objecter ms_handle_connect 0x7f221813f710 2024-03-25T17:44:13.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.929+0000 7f220dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:13.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.940+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154059104 0 0) 0x7f22181396e0 con 0x7f22181430c0 2024-03-25T17:44:13.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.940+0000 7f220dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:44:13.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.940+0000 7f220dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:13.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.940+0000 7f220dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:13.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.940+0000 7f220dffb640 10 monclient(hunting): my global_id is 4464 2024-03-25T17:44:13.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.940+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21fc004eb0 con 0x7f22181430c0 2024-03-25T17:44:13.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.940+0000 7f220dffb640 10 client.?.objecter ms_handle_connect 0x7f22180a98f0 2024-03-25T17:44:13.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.940+0000 7f220dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:13.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.943+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58984473 0 0) 0x7f2218138530 con 0x7f221813f710 2024-03-25T17:44:13.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.943+0000 7f220dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:44:13.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.943+0000 7f220dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:13.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.943+0000 7f220dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:13.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.943+0000 7f220dffb640 10 monclient(hunting): my global_id is 4450 2024-03-25T17:44:13.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.943+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22181396e0 con 0x7f221813f710 2024-03-25T17:44:13.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.955+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464998580 0 0) 0x7f22181399c0 con 0x7f22180a98f0 2024-03-25T17:44:13.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.955+0000 7f220dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:44:13.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.955+0000 7f220dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:13.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.955+0000 7f220dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:13.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.955+0000 7f220dffb640 10 monclient(hunting): my global_id is 4451 2024-03-25T17:44:13.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.955+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2218138530 con 0x7f22180a98f0 2024-03-25T17:44:13.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 648908698 0 0) 0x7f2218138530 con 0x7f22180a98f0 2024-03-25T17:44:13.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22181399c0 con 0x7f22180a98f0 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f22140018f0 con 0x7f22180a98f0 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 10 monclient(hunting): dump: 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:44:13.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902681332 0 0) 0x7f22181396e0 con 0x7f221813f710 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2218138530 con 0x7f221813f710 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2208001870 con 0x7f221813f710 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.968+0000 7f220dffb640 10 monclient(hunting): dump: 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:13.970 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:13.971 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:13.971 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:13.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.971+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 373189404 0 0) 0x7f21fc004eb0 con 0x7f22181430c0 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.971+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22181396e0 con 0x7f22181430c0 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.971+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2210001ec0 con 0x7f22181430c0 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.971+0000 7f220dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.971+0000 7f220dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.971+0000 7f220dffb640 10 monclient(hunting): dump: 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:13.972 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:13.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2881803283 0 0) 0x7f22181399c0 con 0x7f22180a98f0 2024-03-25T17:44:13.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 >> v1:172.21.15.111:6789/0 conn(0x7f221813f710 legacy=0x7f22181da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:13.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 >> v1:172.21.15.44:6789/0 conn(0x7f22181430c0 legacy=0x7f22181e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:13.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:44:13.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 1 monclient: found mon.c 2024-03-25T17:44:13.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218138a50 con 0x7f22180a98f0 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 10 monclient: _finish_auth 0 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f220dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 5 monclient: authenticate success, global_id 4451 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 10 monclient: _renew_subs 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22181380a0 con 0x7f22180a98f0 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 10 monclient: _renew_subs 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22181db140 con 0x7f22180a98f0 2024-03-25T17:44:13.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.982+0000 7f221f298640 1 librados: init done 2024-03-25T17:44:13.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.983+0000 7f221f298640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:13.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.983+0000 7f221f298640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:13.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.983+0000 7f221f298640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:44:13.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:13.983+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2218138ca0 con 0x7f22180a98f0 2024-03-25T17:44:14.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.290+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22140035f0 con 0x7f22180a98f0 2024-03-25T17:44:14.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.290+0000 7f220dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.290+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f2214001d80 con 0x7f22180a98f0 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.290+0000 7f220dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.290+0000 7f220dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.290+0000 7f220dffb640 10 monclient: dump: 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.290+0000 7f220f7fe640 4 set_mon_vals no callback set 2024-03-25T17:44:14.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.291+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f2214019690 con 0x7f22180a98f0 2024-03-25T17:44:14.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.291+0000 7f221c833640 10 monclient: get_auth_request con 0x7f21fc06e0f0 auth_method 0 2024-03-25T17:44:14.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.291+0000 7f220dffb640 10 client.4451.objecter ms_handle_connect 0x7f21fc06e0f0 2024-03-25T17:44:14.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.298+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f2214053340 con 0x7f22180a98f0 2024-03-25T17:44:14.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.298+0000 7f220dffb640 10 client.4451.objecter ms_dispatch 0x7f2218000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:44:14.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.298+0000 7f220dffb640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:44:14.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.298+0000 7f220dffb640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:44:14.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.298+0000 7f220dffb640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T17:44:14.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.298+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f221404ec80 con 0x7f22180a98f0 2024-03-25T17:44:14.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.298+0000 7f220dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:14.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.298+0000 7f220dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:44:14.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.536+0000 7f221f298640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:14.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.536+0000 7f221f298640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:14.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.536+0000 7f221f298640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:44:14.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.536+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2218138720 con 0x7f22180a98f0 2024-03-25T17:44:14.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.562+0000 7f220dffb640 1 -- 172.21.15.44:0/3560856471 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f2214050d80 con 0x7f22180a98f0 2024-03-25T17:44:14.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.562+0000 7f220dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:14.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.562+0000 7f220dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:44:14.563 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:44:14.563 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.564+0000 7f221f298640 10 librados: watch_flush enter 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.564+0000 7f221f298640 10 librados: watch_flush exit 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.564+0000 7f221f298640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 10 client.4451.objecter successfully canceled tick 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 >> v1:172.21.15.44:6800/3257043132 conn(0x7f21fc06e0f0 legacy=0x7f21fc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 10 monclient: shutdown 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 >> v1:172.21.15.44:6790/0 conn(0x7f22180a98f0 legacy=0x7f2218137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 shutdown_connections 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 >> 172.21.15.44:0/3560856471 conn(0x7f22180b7ea0 msgr2=0x7f2218074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 shutdown_connections 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 1 -- 172.21.15.44:0/3560856471 wait complete. 2024-03-25T17:44:14.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:14.565+0000 7f221f298640 1 librados: shutdown 2024-03-25T17:44:14.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:44:18.006 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:35.362 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:18.011 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:35.367 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:18.052 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:35.408 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:18.070 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:35.390 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:18.080 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:35.436 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:18.081 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:35.401 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:18.095 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:35.451 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:18.098 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:35.418 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:18.112 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:35.468 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:21.385 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.502+0000 7fa5b0e06640 10 monclient: get_monmap_and_config 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.502+0000 7fa5b0e06640 10 monclient: build_initial_monmap 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.502+0000 7fa5b0e06640 10 monclient: monmap: 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:44:21.503754+0000 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:44:21.503754+0000 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:21.503 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:21.504 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:21.504 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:44:21.504 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:44:21.504 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:21.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.503+0000 7fa5b0e06640 1 Processor -- start 2024-03-25T17:44:21.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.503+0000 7fa5b0e06640 1 -- start start 2024-03-25T17:44:21.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.503+0000 7fa5b0e06640 10 monclient: init 2024-03-25T17:44:21.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:21.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:44:21.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ac0b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:21.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ac0b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:21.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ac14be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:21.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0af940 con 0x7fa5ac14be90 2024-03-25T17:44:21.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0b0a70 con 0x7fa5ac0b1730 2024-03-25T17:44:21.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0b0d50 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:21.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.504+0000 7fa5b0e06640 10 monclient(hunting): authenticate will time out at 6264.731890s 2024-03-25T17:44:21.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.515+0000 7fa5a1d9b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa5ac0b1730 0x7fa5ac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45700/0 (socket says 172.21.15.44:45700) 2024-03-25T17:44:21.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.515+0000 7fa5a1d9b640 1 -- 172.21.15.44:0/2594524897 learned_addr learned my addr 172.21.15.44:0/2594524897 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:21.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.538+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739230864 0 0) 0x7fa5ac0b0a70 con 0x7fa5ac0b1730 2024-03-25T17:44:21.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.538+0000 7fa5a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:44:21.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.538+0000 7fa5a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:21.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.538+0000 7fa5a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:21.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.538+0000 7fa5a9d9b640 10 monclient(hunting): my global_id is 4453 2024-03-25T17:44:21.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.538+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa598003520 con 0x7fa5ac0b1730 2024-03-25T17:44:21.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.545+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296707157 0 0) 0x7fa5ac0b0d50 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.545+0000 7fa5a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:44:21.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.545+0000 7fa5a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:21.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.545+0000 7fa5a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:21.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.545+0000 7fa5a9d9b640 10 monclient(hunting): my global_id is 4454 2024-03-25T17:44:21.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.545+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac0b0a70 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765668802 0 0) 0x7fa598003520 con 0x7fa5ac0b1730 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 >> v1:172.21.15.44:6790/0 conn(0x7fa5ac0b4d60 legacy=0x7fa5ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 >> v1:172.21.15.44:6789/0 conn(0x7fa5ac14be90 legacy=0x7fa5ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 1 monclient: found mon.noname-c 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac0b1030 con 0x7fa5ac0b1730 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 10 monclient: _finish_auth 0 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa594001690 con 0x7fa5ac0b1730 2024-03-25T17:44:21.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5b0e06640 5 monclient: authenticate success, global_id 4453 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5b0e06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5a9d9b640 10 monclient: dump: 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:21.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.554+0000 7fa5b0e06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:21.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.569+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5940017f0 con 0x7fa5ac0b1730 2024-03-25T17:44:21.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.569+0000 7fa5a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:21.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.569+0000 7fa5a9d9b640 1 -- 172.21.15.44:0/2594524897 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa594001c00 con 0x7fa5ac0b1730 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.569+0000 7fa5a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.569+0000 7fa5a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.569+0000 7fa5a9d9b640 10 monclient: dump: 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:21.570 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:21.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.569+0000 7fa5b0e06640 10 monclient: get_monmap_and_config success 2024-03-25T17:44:21.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.569+0000 7fa5b0e06640 4 set_mon_vals no callback set 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.570+0000 7fa5b0e06640 10 monclient: shutdown 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.570+0000 7fa5b0e06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.570+0000 7fa5b0e06640 1 -- 172.21.15.44:0/2594524897 >> v1:172.21.15.111:6789/0 conn(0x7fa5ac0b1730 legacy=0x7fa5ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.570+0000 7fa5b0e06640 1 -- 172.21.15.44:0/2594524897 shutdown_connections 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.570+0000 7fa5b0e06640 1 -- 172.21.15.44:0/2594524897 >> 172.21.15.44:0/2594524897 conn(0x7fa5ac0617a0 msgr2=0x7fa5ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.570+0000 7fa5b0e06640 1 -- 172.21.15.44:0/2594524897 shutdown_connections 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.570+0000 7fa5b0e06640 1 -- 172.21.15.44:0/2594524897 wait complete. 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 10 monclient: build_initial_monmap 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 10 monclient: monmap: 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:21.572 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 1 librados: starting msgr at 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 1 librados: starting objecter 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 1 Processor -- start 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 1 -- start start 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 1 librados: setting wanted keys 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 1 librados: calling monclient init 2024-03-25T17:44:21.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.571+0000 7fa5b0e06640 10 monclient: init 2024-03-25T17:44:21.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ac0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ac0b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ac14be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac1d83d0 con 0x7fa5ac14be90 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac1d77a0 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac138270 con 0x7fa5ac0b1730 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:21.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.572+0000 7fa5b0e06640 10 monclient(hunting): authenticate will time out at 6264.800339s 2024-03-25T17:44:21.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.581+0000 7fa5a1d9b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa5ac0b4d60 0x7fa5ac1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45738/0 (socket says 172.21.15.44:45738) 2024-03-25T17:44:21.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.581+0000 7fa5a1d9b640 1 -- 172.21.15.44:0/3690019241 learned_addr learned my addr 172.21.15.44:0/3690019241 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:21.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.582+0000 7fa5a37fe640 10 client.?.objecter ms_handle_connect 0x7fa5ac0b4d60 2024-03-25T17:44:21.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.582+0000 7fa5a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:21.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.583+0000 7fa5a37fe640 10 client.?.objecter ms_handle_connect 0x7fa5ac14be90 2024-03-25T17:44:21.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.583+0000 7fa5a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:21.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.585+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670573379 0 0) 0x7fa5ac1d77a0 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.585+0000 7fa5a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:44:21.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.585+0000 7fa5a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:21.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.585+0000 7fa5a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:21.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.585+0000 7fa5a37fe640 10 monclient(hunting): my global_id is 4456 2024-03-25T17:44:21.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.585+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa58c004eb0 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.589+0000 7fa5a37fe640 10 client.?.objecter ms_handle_connect 0x7fa5ac0b1730 2024-03-25T17:44:21.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.589+0000 7fa5a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.593+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1556528092 0 0) 0x7fa58c004eb0 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.593+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5ac1d77a0 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.593+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa594001c40 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.593+0000 7fa5a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.593+0000 7fa5a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.593+0000 7fa5a37fe640 10 monclient(hunting): dump: 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:21.594 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:21.595 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:21.595 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:21.595 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:21.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.593+0000 7fa5a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:44:21.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.602+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948839253 0 0) 0x7fa5ac138270 con 0x7fa5ac0b1730 2024-03-25T17:44:21.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.602+0000 7fa5a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:44:21.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.602+0000 7fa5a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:21.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.602+0000 7fa5a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:21.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.602+0000 7fa5a37fe640 10 monclient(hunting): my global_id is 4457 2024-03-25T17:44:21.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.602+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa58c004eb0 con 0x7fa5ac0b1730 2024-03-25T17:44:21.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.603+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447435534 0 0) 0x7fa5ac1d83d0 con 0x7fa5ac14be90 2024-03-25T17:44:21.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.603+0000 7fa5a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:44:21.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.603+0000 7fa5a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:21.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.603+0000 7fa5a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:21.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.603+0000 7fa5a37fe640 10 monclient(hunting): my global_id is 4470 2024-03-25T17:44:21.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.603+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac138270 con 0x7fa5ac14be90 2024-03-25T17:44:21.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.611+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2437083911 0 0) 0x7fa5ac138270 con 0x7fa5ac14be90 2024-03-25T17:44:21.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.611+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5ac1d83d0 con 0x7fa5ac14be90 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3507600097 0 0) 0x7fa5ac1d77a0 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 >> v1:172.21.15.44:6790/0 conn(0x7fa5ac0b1730 legacy=0x7fa5ac1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 >> v1:172.21.15.44:6789/0 conn(0x7fa5ac14be90 legacy=0x7fa5ac142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 1 monclient: found mon.b 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac140d00 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 10 monclient: _finish_auth 0 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 5 monclient: authenticate success, global_id 4456 2024-03-25T17:44:21.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 10 monclient: _renew_subs 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac138a80 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 10 monclient: _renew_subs 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ac143640 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.613+0000 7fa5b0e06640 1 librados: init done 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.614+0000 7fa5b0e06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:21.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.614+0000 7fa5b0e06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:21.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.614+0000 7fa5b0e06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:21.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.614+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ac140ef0 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.888+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa594004a90 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.888+0000 7fa5a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.888+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa594004c80 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.888+0000 7fa5a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.888+0000 7fa5a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.888+0000 7fa5a37fe640 10 monclient: dump: 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:21.889 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:21.890 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:21.890 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:21.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.888+0000 7fa5a959a640 4 set_mon_vals no callback set 2024-03-25T17:44:21.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.889+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fa594004ea0 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.889+0000 7fa5aa59c640 10 monclient: get_auth_request con 0x7fa58c06e150 auth_method 0 2024-03-25T17:44:21.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.890+0000 7fa5a37fe640 10 client.4456.objecter ms_handle_connect 0x7fa58c06e150 2024-03-25T17:44:21.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.894+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fa594054870 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.894+0000 7fa5a37fe640 10 client.4456.objecter ms_dispatch 0x7fa5ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:44:21.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.894+0000 7fa5a37fe640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:44:21.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.894+0000 7fa5a37fe640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:44:21.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.894+0000 7fa5a37fe640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T17:44:21.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.894+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa594050380 con 0x7fa5ac0b4d60 2024-03-25T17:44:21.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.894+0000 7fa5a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:21.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:21.894+0000 7fa5a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:44:22.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.126+0000 7fa5b0e06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:22.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.126+0000 7fa5b0e06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:22.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.126+0000 7fa5b0e06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:22.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.126+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5ac0afbd0 con 0x7fa5ac0b4d60 2024-03-25T17:44:22.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.160+0000 7fa5a37fe640 1 -- 172.21.15.44:0/3690019241 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fa5940522b0 con 0x7fa5ac0b4d60 2024-03-25T17:44:22.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.160+0000 7fa5a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:22.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.160+0000 7fa5a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:44:22.161 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:44:22.161 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:44:22.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 10 librados: watch_flush enter 2024-03-25T17:44:22.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 10 librados: watch_flush exit 2024-03-25T17:44:22.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T17:44:22.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 10 client.4456.objecter successfully canceled tick 2024-03-25T17:44:22.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 >> v1:172.21.15.44:6800/3257043132 conn(0x7fa58c06e150 legacy=0x7fa58c070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:22.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 10 monclient: shutdown 2024-03-25T17:44:22.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:22.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 >> v1:172.21.15.111:6789/0 conn(0x7fa5ac0b4d60 legacy=0x7fa5ac1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:22.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 shutdown_connections 2024-03-25T17:44:22.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 >> 172.21.15.44:0/3690019241 conn(0x7fa5ac0617a0 msgr2=0x7fa5ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:22.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 shutdown_connections 2024-03-25T17:44:22.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.162+0000 7fa5b0e06640 1 -- 172.21.15.44:0/3690019241 wait complete. 2024-03-25T17:44:22.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:22.163+0000 7fa5b0e06640 1 librados: shutdown 2024-03-25T17:44:22.176 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:44:28.983 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.098+0000 7fbbb0463640 10 monclient: get_monmap_and_config 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.098+0000 7fbbb0463640 10 monclient: build_initial_monmap 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.098+0000 7fbbb0463640 10 monclient: monmap: 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:44:29.099399+0000 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:44:29.099399+0000 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:44:29.099 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:44:29.100 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.099+0000 7fbbb0463640 1 Processor -- start 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.099+0000 7fbbb0463640 1 -- start start 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.099+0000 7fbbb0463640 10 monclient: init 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 10 monclient(hunting): picked mon.noname-a con 0x7fbba806c380 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 10 monclient(hunting): picked mon.noname-c con 0x7fbba806d670 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 10 monclient(hunting): picked mon.noname-b con 0x7fbba80af8f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:29.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba806c9a0 con 0x7fbba806c380 2024-03-25T17:44:29.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba80ada90 con 0x7fbba806d670 2024-03-25T17:44:29.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba80b0d10 con 0x7fbba80af8f0 2024-03-25T17:44:29.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:29.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.100+0000 7fbbb0463640 10 monclient(hunting): authenticate will time out at 6272.328156s 2024-03-25T17:44:29.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.102+0000 7fbbad9fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbba806d670 0x7fbba80ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:50502/0 (socket says 172.21.15.44:50502) 2024-03-25T17:44:29.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.102+0000 7fbbad9fe640 1 -- 172.21.15.44:0/1196700344 learned_addr learned my addr 172.21.15.44:0/1196700344 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:29.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.113+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974355226 0 0) 0x7fbba80ada90 con 0x7fbba806d670 2024-03-25T17:44:29.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.113+0000 7fbbad1fd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:29.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.113+0000 7fbbad1fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:29.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.113+0000 7fbbad1fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:29.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.113+0000 7fbbad1fd640 10 monclient(hunting): my global_id is 4459 2024-03-25T17:44:29.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.113+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb94003520 con 0x7fbba806d670 2024-03-25T17:44:29.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.132+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197454002 0 0) 0x7fbba806c9a0 con 0x7fbba806c380 2024-03-25T17:44:29.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.132+0000 7fbbad1fd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:29.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.132+0000 7fbbad1fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:29.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.132+0000 7fbbad1fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:29.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.132+0000 7fbbad1fd640 10 monclient(hunting): my global_id is 4473 2024-03-25T17:44:29.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.132+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba80ada90 con 0x7fbba806c380 2024-03-25T17:44:29.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1221401363 0 0) 0x7fbb94003520 con 0x7fbba806d670 2024-03-25T17:44:29.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 >> v1:172.21.15.44:6790/0 conn(0x7fbba80af8f0 legacy=0x7fbba80aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:29.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 >> v1:172.21.15.44:6789/0 conn(0x7fbba806c380 legacy=0x7fbba80b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:29.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 10 monclient: _finish_hunting 0 2024-03-25T17:44:29.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 1 monclient: found mon.noname-c 2024-03-25T17:44:29.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba80add00 con 0x7fbba806d670 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 10 monclient: _finish_auth 0 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 10 monclient: _check_auth_tickets 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbb98001690 con 0x7fbba806d670 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbad1fd640 10 monclient: dump: 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.146 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:29.147 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:29.147 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:29.147 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:29.147 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:29.147 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:29.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbb0463640 5 monclient: authenticate success, global_id 4459 2024-03-25T17:44:29.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.144+0000 7fbbb0463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:29.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.158+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb980017f0 con 0x7fbba806d670 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.158+0000 7fbbad1fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.158+0000 7fbbad1fd640 1 -- 172.21.15.44:0/1196700344 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbb98001b80 con 0x7fbba806d670 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.158+0000 7fbbad1fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.158+0000 7fbbad1fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.158+0000 7fbbad1fd640 10 monclient: dump: 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:29.159 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:29.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.158+0000 7fbbb0463640 10 monclient: get_monmap_and_config success 2024-03-25T17:44:29.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.158+0000 7fbbb0463640 4 set_mon_vals no callback set 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.159+0000 7fbbb0463640 10 monclient: shutdown 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.159+0000 7fbbb0463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.159+0000 7fbbb0463640 1 -- 172.21.15.44:0/1196700344 >> v1:172.21.15.111:6789/0 conn(0x7fbba806d670 legacy=0x7fbba80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.159+0000 7fbbb0463640 1 -- 172.21.15.44:0/1196700344 shutdown_connections 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.159+0000 7fbbb0463640 1 -- 172.21.15.44:0/1196700344 >> 172.21.15.44:0/1196700344 conn(0x7fbba80b5c00 msgr2=0x7fbba80b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.159+0000 7fbbb0463640 1 -- 172.21.15.44:0/1196700344 shutdown_connections 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.159+0000 7fbbb0463640 1 -- 172.21.15.44:0/1196700344 wait complete. 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 10 monclient: build_initial_monmap 2024-03-25T17:44:29.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 10 monclient: monmap: 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 1 librados: starting msgr at 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 1 librados: starting objecter 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 1 Processor -- start 2024-03-25T17:44:29.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 1 -- start start 2024-03-25T17:44:29.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 1 librados: setting wanted keys 2024-03-25T17:44:29.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 1 librados: calling monclient init 2024-03-25T17:44:29.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.160+0000 7fbbb0463640 10 monclient: init 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 10 monclient(hunting): picked mon.noname-b con 0x7fbba806c380 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 10 monclient(hunting): picked mon.noname-a con 0x7fbba806d670 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 10 monclient(hunting): picked mon.noname-c con 0x7fbba80af8f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba8145040 con 0x7fbba806d670 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba8145320 con 0x7fbba806c380 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba8145600 con 0x7fbba80af8f0 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:29.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.161+0000 7fbbb0463640 10 monclient(hunting): authenticate will time out at 6272.389294s 2024-03-25T17:44:29.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.171+0000 7fbbad9fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbba806d670 0x7fbba81490d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52684/0 (socket says 172.21.15.44:52684) 2024-03-25T17:44:29.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.171+0000 7fbbad9fe640 1 -- 172.21.15.44:0/1770237886 learned_addr learned my addr 172.21.15.44:0/1770237886 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:29.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.172+0000 7fbb9effd640 10 client.?.objecter ms_handle_connect 0x7fbba806d670 2024-03-25T17:44:29.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.172+0000 7fbb9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:29.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.181+0000 7fbb9effd640 10 client.?.objecter ms_handle_connect 0x7fbba806c380 2024-03-25T17:44:29.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.181+0000 7fbb9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:29.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.183+0000 7fbb9effd640 10 client.?.objecter ms_handle_connect 0x7fbba80af8f0 2024-03-25T17:44:29.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.183+0000 7fbb9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:29.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.186+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813320843 0 0) 0x7fbba8145320 con 0x7fbba806c380 2024-03-25T17:44:29.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.186+0000 7fbb9effd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:29.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.186+0000 7fbb9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:29.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.186+0000 7fbb9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:29.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.186+0000 7fbb9effd640 10 monclient(hunting): my global_id is 4462 2024-03-25T17:44:29.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.186+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb8c004eb0 con 0x7fbba806c380 2024-03-25T17:44:29.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.194+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011374175 0 0) 0x7fbba8145040 con 0x7fbba806d670 2024-03-25T17:44:29.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.194+0000 7fbb9effd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:29.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.194+0000 7fbb9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:29.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.194+0000 7fbb9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:29.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.194+0000 7fbb9effd640 10 monclient(hunting): my global_id is 4476 2024-03-25T17:44:29.195 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.194+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba8145320 con 0x7fbba806d670 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.203+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 627002899 0 0) 0x7fbb8c004eb0 con 0x7fbba806c380 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.203+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbba8145040 con 0x7fbba806c380 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.203+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbba4001930 con 0x7fbba806c380 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.203+0000 7fbb9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.203+0000 7fbb9effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.203+0000 7fbb9effd640 10 monclient(hunting): dump: 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.205 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:29.205 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:29.205 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:29.205 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:29.205 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:29.205 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:29.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.203+0000 7fbb9effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.204+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1991788418 0 0) 0x7fbba8145320 con 0x7fbba806d670 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.204+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb8c004eb0 con 0x7fbba806d670 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.204+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbb98001b00 con 0x7fbba806d670 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.204+0000 7fbb9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.204+0000 7fbb9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.204+0000 7fbb9effd640 10 monclient(hunting): dump: 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:29.206 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.206+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682598517 0 0) 0x7fbba8145600 con 0x7fbba80af8f0 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.206+0000 7fbb9effd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.206+0000 7fbb9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.206+0000 7fbb9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.206+0000 7fbb9effd640 10 monclient(hunting): my global_id is 4463 2024-03-25T17:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.206+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba8145320 con 0x7fbba80af8f0 2024-03-25T17:44:29.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3159372554 0 0) 0x7fbba8145040 con 0x7fbba806c380 2024-03-25T17:44:29.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 >> v1:172.21.15.44:6790/0 conn(0x7fbba80af8f0 legacy=0x7fbba81e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:29.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 >> v1:172.21.15.44:6789/0 conn(0x7fbba806d670 legacy=0x7fbba81490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:29.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 10 monclient: _finish_hunting 0 2024-03-25T17:44:29.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 1 monclient: found mon.b 2024-03-25T17:44:29.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:29.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba8143a50 con 0x7fbba806c380 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 10 monclient: _finish_auth 0 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 10 monclient: _check_auth_tickets 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbb9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 5 monclient: authenticate success, global_id 4462 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 10 monclient: _renew_subs 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba8142fa0 con 0x7fbba806c380 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 10 monclient: _renew_subs 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba80b5210 con 0x7fbba806c380 2024-03-25T17:44:29.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.217+0000 7fbbb0463640 1 librados: init done 2024-03-25T17:44:29.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.218+0000 7fbbb0463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:29.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.218+0000 7fbbb0463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:29.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.218+0000 7fbbb0463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:29.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.218+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba80627b0 con 0x7fbba806c380 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.533+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbba4001a90 con 0x7fbba806c380 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.533+0000 7fbb9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.533+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbba40038c0 con 0x7fbba806c380 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.533+0000 7fbb9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.533+0000 7fbb9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.533+0000 7fbbac9fc640 4 set_mon_vals no callback set 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.533+0000 7fbb9effd640 10 monclient: dump: 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:29.534 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:29.535 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:29.535 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:29.535 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:29.535 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:29.535 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:29.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.533+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fbba4019950 con 0x7fbba806c380 2024-03-25T17:44:29.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.534+0000 7fbbad9fe640 10 monclient: get_auth_request con 0x7fbb8c06df00 auth_method 0 2024-03-25T17:44:29.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.534+0000 7fbb9effd640 10 client.4462.objecter ms_handle_connect 0x7fbb8c06df00 2024-03-25T17:44:29.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.541+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fbba4053640 con 0x7fbba806c380 2024-03-25T17:44:29.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.541+0000 7fbb9effd640 10 client.4462.objecter ms_dispatch 0x7fbba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:44:29.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.541+0000 7fbb9effd640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:44:29.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.541+0000 7fbb9effd640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:44:29.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.541+0000 7fbb9effd640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T17:44:29.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.542+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbba404ef80 con 0x7fbba806c380 2024-03-25T17:44:29.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.542+0000 7fbb9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:29.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.542+0000 7fbb9effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:44:29.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.798+0000 7fbbb0463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:29.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.798+0000 7fbbb0463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:29.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.798+0000 7fbbb0463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:29.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.798+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbba8070170 con 0x7fbba806c380 2024-03-25T17:44:29.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.828+0000 7fbb9effd640 1 -- 172.21.15.44:0/1770237886 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fbba4051080 con 0x7fbba806c380 2024-03-25T17:44:29.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.829+0000 7fbb9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:29.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.829+0000 7fbb9effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:44:29.830 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:44:29.830 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:44:29.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.831+0000 7fbbb0463640 10 librados: watch_flush enter 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.831+0000 7fbbb0463640 10 librados: watch_flush exit 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.831+0000 7fbbb0463640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.831+0000 7fbbb0463640 10 client.4462.objecter successfully canceled tick 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 >> v1:172.21.15.44:6800/3257043132 conn(0x7fbb8c06df00 legacy=0x7fbb8c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 10 monclient: shutdown 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 >> v1:172.21.15.111:6789/0 conn(0x7fbba806c380 legacy=0x7fbba8143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 shutdown_connections 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 >> 172.21.15.44:0/1770237886 conn(0x7fbba80b5c00 msgr2=0x7fbba80b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 shutdown_connections 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 1 -- 172.21.15.44:0/1770237886 wait complete. 2024-03-25T17:44:29.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:29.832+0000 7fbbb0463640 1 librados: shutdown 2024-03-25T17:44:29.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:44:36.653 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:44:36.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.769+0000 7f1e0a9bc640 10 monclient: get_monmap_and_config 2024-03-25T17:44:36.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.769+0000 7f1e0a9bc640 10 monclient: build_initial_monmap 2024-03-25T17:44:36.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.769+0000 7f1e0a9bc640 10 monclient: monmap: 2024-03-25T17:44:36.770 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:44:36.770821+0000 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:44:36.770821+0000 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:44:36.771 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:36.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.770+0000 7f1e0a9bc640 1 Processor -- start 2024-03-25T17:44:36.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.770+0000 7f1e0a9bc640 1 -- start start 2024-03-25T17:44:36.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.770+0000 7f1e0a9bc640 10 monclient: init 2024-03-25T17:44:36.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 10 monclient(hunting): picked mon.noname-c con 0x7f1e040a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 10 monclient(hunting): picked mon.noname-a con 0x7f1e0413f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 10 monclient(hunting): picked mon.noname-b con 0x7f1e041430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e04142520 con 0x7f1e0413f710 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e040b28a0 con 0x7f1e040a98f0 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e040b2a30 con 0x7f1e041430c0 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:36.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.771+0000 7f1e0a9bc640 10 monclient(hunting): authenticate will time out at 6279.999538s 2024-03-25T17:44:36.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.780+0000 7f1e037fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1e0413f710 0x7f1e041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48980/0 (socket says 172.21.15.44:48980) 2024-03-25T17:44:36.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.780+0000 7f1e037fe640 1 -- 172.21.15.44:0/3271424111 learned_addr learned my addr 172.21.15.44:0/3271424111 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:36.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.787+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873652738 0 0) 0x7f1e040b28a0 con 0x7f1e040a98f0 2024-03-25T17:44:36.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.788+0000 7f1e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:36.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.788+0000 7f1e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:36.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.788+0000 7f1e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:36.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.788+0000 7f1e02ffd640 10 monclient(hunting): my global_id is 4465 2024-03-25T17:44:36.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.788+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec003520 con 0x7f1e040a98f0 2024-03-25T17:44:36.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.793+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116904771 0 0) 0x7f1e04142520 con 0x7f1e0413f710 2024-03-25T17:44:36.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.793+0000 7f1e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:36.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.793+0000 7f1e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:36.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.793+0000 7f1e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:36.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.793+0000 7f1e02ffd640 10 monclient(hunting): my global_id is 4479 2024-03-25T17:44:36.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.793+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e040b28a0 con 0x7f1e0413f710 2024-03-25T17:44:36.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4109379279 0 0) 0x7f1dec003520 con 0x7f1e040a98f0 2024-03-25T17:44:36.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 >> v1:172.21.15.44:6790/0 conn(0x7f1e041430c0 legacy=0x7f1e0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:36.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 >> v1:172.21.15.44:6789/0 conn(0x7f1e0413f710 legacy=0x7f1e041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:36.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:44:36.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 1 monclient: found mon.noname-c 2024-03-25T17:44:36.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:36.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e040b44c0 con 0x7f1e040a98f0 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 10 monclient: _finish_auth 0 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1df4001690 con 0x7f1e040a98f0 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e0a9bc640 5 monclient: authenticate success, global_id 4465 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e0a9bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e02ffd640 10 monclient: dump: 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.799 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:36.800 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:36.800 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:36.800 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:36.800 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:36.800 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:36.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.797+0000 7f1e0a9bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:36.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.814+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1df40017f0 con 0x7f1e040a98f0 2024-03-25T17:44:36.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.814+0000 7f1e02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.814+0000 7f1e02ffd640 1 -- 172.21.15.44:0/3271424111 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1df4001c00 con 0x7f1e040a98f0 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.814+0000 7f1e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.814+0000 7f1e02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.814+0000 7f1e02ffd640 10 monclient: dump: 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:36.815 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:36.816 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:36.816 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:36.816 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:36.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.814+0000 7f1e0a9bc640 10 monclient: get_monmap_and_config success 2024-03-25T17:44:36.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.814+0000 7f1e0a9bc640 4 set_mon_vals no callback set 2024-03-25T17:44:36.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.815+0000 7f1e0a9bc640 10 monclient: shutdown 2024-03-25T17:44:36.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.815+0000 7f1e0a9bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:36.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.815+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/3271424111 >> v1:172.21.15.111:6789/0 conn(0x7f1e040a98f0 legacy=0x7f1e040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.815+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/3271424111 shutdown_connections 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.815+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/3271424111 >> 172.21.15.44:0/3271424111 conn(0x7f1e040b7ea0 msgr2=0x7f1e040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.815+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/3271424111 shutdown_connections 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.816+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/3271424111 wait complete. 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.816+0000 7f1e0a9bc640 10 monclient: build_initial_monmap 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.816+0000 7f1e0a9bc640 10 monclient: monmap: 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:36.818 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 1 librados: starting msgr at 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 1 librados: starting objecter 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 1 Processor -- start 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 1 -- start start 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 1 librados: setting wanted keys 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 1 librados: calling monclient init 2024-03-25T17:44:36.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 10 monclient: init 2024-03-25T17:44:36.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:36.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:44:36.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 10 monclient(hunting): picked mon.noname-c con 0x7f1e040a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:36.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.817+0000 7f1e0a9bc640 10 monclient(hunting): picked mon.noname-b con 0x7f1e0413f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:36.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.818+0000 7f1e0a9bc640 10 monclient(hunting): picked mon.noname-a con 0x7f1e041430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:36.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.818+0000 7f1e0a9bc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e04137ab0 con 0x7f1e041430c0 2024-03-25T17:44:36.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.818+0000 7f1e0a9bc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e04138bd0 con 0x7f1e0413f710 2024-03-25T17:44:36.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.818+0000 7f1e0a9bc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e04138e40 con 0x7f1e040a98f0 2024-03-25T17:44:36.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.818+0000 7f1e0a9bc640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:36.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.818+0000 7f1e0a9bc640 10 monclient(hunting): authenticate will time out at 6280.045845s 2024-03-25T17:44:36.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.828+0000 7f1e03fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f1e040a98f0 0x7f1e04137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:58800/0 (socket says 172.21.15.44:58800) 2024-03-25T17:44:36.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.828+0000 7f1e03fff640 1 -- 172.21.15.44:0/4058663960 learned_addr learned my addr 172.21.15.44:0/4058663960 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:36.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.831+0000 7f1e00ff9640 10 client.?.objecter ms_handle_connect 0x7f1e0413f710 2024-03-25T17:44:36.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.831+0000 7f1e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:36.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.832+0000 7f1e00ff9640 10 client.?.objecter ms_handle_connect 0x7f1e040a98f0 2024-03-25T17:44:36.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.832+0000 7f1e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:36.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.833+0000 7f1e00ff9640 10 client.?.objecter ms_handle_connect 0x7f1e041430c0 2024-03-25T17:44:36.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.833+0000 7f1e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:36.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.841+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006123265 0 0) 0x7f1e04138bd0 con 0x7f1e0413f710 2024-03-25T17:44:36.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.841+0000 7f1e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:44:36.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.841+0000 7f1e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:36.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.841+0000 7f1e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:36.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.841+0000 7f1e00ff9640 10 monclient(hunting): my global_id is 4468 2024-03-25T17:44:36.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.841+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df0004eb0 con 0x7f1e0413f710 2024-03-25T17:44:36.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.846+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955810191 0 0) 0x7f1e04138e40 con 0x7f1e040a98f0 2024-03-25T17:44:36.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.846+0000 7f1e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:44:36.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.846+0000 7f1e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:36.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.846+0000 7f1e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:36.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.846+0000 7f1e00ff9640 10 monclient(hunting): my global_id is 4469 2024-03-25T17:44:36.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.846+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e04138bd0 con 0x7f1e040a98f0 2024-03-25T17:44:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.847+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182633842 0 0) 0x7f1e04137ab0 con 0x7f1e041430c0 2024-03-25T17:44:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.847+0000 7f1e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:44:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.847+0000 7f1e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.847+0000 7f1e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.847+0000 7f1e00ff9640 10 monclient(hunting): my global_id is 4482 2024-03-25T17:44:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.847+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e04138e40 con 0x7f1e041430c0 2024-03-25T17:44:36.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.866+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2327878282 0 0) 0x7f1df0004eb0 con 0x7f1e0413f710 2024-03-25T17:44:36.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.866+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e04137ab0 con 0x7f1e0413f710 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.866+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1df8001930 con 0x7f1e0413f710 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.866+0000 7f1e00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.866+0000 7f1e00ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.866+0000 7f1e00ff9640 10 monclient(hunting): dump: 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:36.867 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:36.868 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:36.868 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:36.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.866+0000 7f1e00ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:44:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.875+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 977106951 0 0) 0x7f1e04138e40 con 0x7f1e041430c0 2024-03-25T17:44:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.875+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1df0004eb0 con 0x7f1e041430c0 2024-03-25T17:44:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.875+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1dfc001ad0 con 0x7f1e041430c0 2024-03-25T17:44:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.875+0000 7f1e00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.875+0000 7f1e00ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.875+0000 7f1e00ff9640 10 monclient(hunting): dump: 2024-03-25T17:44:36.875 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:36.876 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.879+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2920118730 0 0) 0x7f1e04138bd0 con 0x7f1e040a98f0 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.879+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e04138e40 con 0x7f1e040a98f0 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.879+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1df4001b40 con 0x7f1e040a98f0 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.879+0000 7f1e00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.879+0000 7f1e00ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.879+0000 7f1e00ff9640 10 monclient(hunting): dump: 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:36.880 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:36.881 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:36.881 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:36.881 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:36.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 854829152 0 0) 0x7f1e04137ab0 con 0x7f1e0413f710 2024-03-25T17:44:36.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 >> v1:172.21.15.44:6790/0 conn(0x7f1e040a98f0 legacy=0x7f1e04137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:36.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 >> v1:172.21.15.44:6789/0 conn(0x7f1e041430c0 legacy=0x7f1e041e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:36.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:44:36.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 1 monclient: found mon.b 2024-03-25T17:44:36.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:36.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e04139120 con 0x7f1e0413f710 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 10 monclient: _finish_auth 0 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 5 monclient: authenticate success, global_id 4468 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 10 monclient: _renew_subs 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e04139400 con 0x7f1e0413f710 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 10 monclient: _renew_subs 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e040b7880 con 0x7f1e0413f710 2024-03-25T17:44:36.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.888+0000 7f1e0a9bc640 1 librados: init done 2024-03-25T17:44:36.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.890+0000 7f1e0a9bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:36.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.890+0000 7f1e0a9bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:36.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.890+0000 7f1e0a9bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:36.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.890+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e04137810 con 0x7f1e0413f710 2024-03-25T17:44:36.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.931+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1df80047e0 con 0x7f1e0413f710 2024-03-25T17:44:36.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.931+0000 7f1e00ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:36.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.931+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1df8004af0 con 0x7f1e0413f710 2024-03-25T17:44:36.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.931+0000 7f1e00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.931+0000 7f1e00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.931+0000 7f1e00ff9640 10 monclient: dump: 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:36.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.931+0000 7f1e027fc640 4 set_mon_vals no callback set 2024-03-25T17:44:36.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.932+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f1df801ab00 con 0x7f1e0413f710 2024-03-25T17:44:36.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.932+0000 7f1e03fff640 10 monclient: get_auth_request con 0x7f1df006e0f0 auth_method 0 2024-03-25T17:44:36.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:36.933+0000 7f1e00ff9640 10 client.4468.objecter ms_handle_connect 0x7f1df006e0f0 2024-03-25T17:44:37.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.220+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f1df80547c0 con 0x7f1e0413f710 2024-03-25T17:44:37.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.220+0000 7f1e00ff9640 10 client.4468.objecter ms_dispatch 0x7f1e04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:44:37.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.220+0000 7f1e00ff9640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:44:37.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.220+0000 7f1e00ff9640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:44:37.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.220+0000 7f1e00ff9640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T17:44:37.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.221+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1df8050100 con 0x7f1e0413f710 2024-03-25T17:44:37.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.221+0000 7f1e00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:37.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.221+0000 7f1e00ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:44:37.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.479+0000 7f1e0a9bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:37.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.479+0000 7f1e0a9bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:37.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.479+0000 7f1e0a9bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:37.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.479+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e040627b0 con 0x7f1e0413f710 2024-03-25T17:44:37.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.523+0000 7f1e00ff9640 1 -- 172.21.15.44:0/4058663960 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f1df8052200 con 0x7f1e0413f710 2024-03-25T17:44:37.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.523+0000 7f1e00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:37.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.523+0000 7f1e00ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:44:37.524 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:44:37.524 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:44:37.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 10 librados: watch_flush enter 2024-03-25T17:44:37.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 10 librados: watch_flush exit 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 10 client.4468.objecter successfully canceled tick 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 >> v1:172.21.15.44:6800/3257043132 conn(0x7f1df006e0f0 legacy=0x7f1df00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 10 monclient: shutdown 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 >> v1:172.21.15.111:6789/0 conn(0x7f1e0413f710 legacy=0x7f1e041df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 shutdown_connections 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 >> 172.21.15.44:0/4058663960 conn(0x7f1e040b7ea0 msgr2=0x7f1e04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 shutdown_connections 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 1 -- 172.21.15.44:0/4058663960 wait complete. 2024-03-25T17:44:37.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:37.525+0000 7f1e0a9bc640 1 librados: shutdown 2024-03-25T17:44:37.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:44:38.067 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:55.423 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:38.075 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:55.431 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:38.110 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:55.466 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:38.121 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:55.477 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:38.135 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:55.491 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:38.136 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:55.456 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:38.141 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:55.461 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:38.169 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:55.489 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:38.170 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:55.526 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:44.344 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:44:44.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.452+0000 7fae18dbc640 10 monclient: get_monmap_and_config 2024-03-25T17:44:44.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.452+0000 7fae18dbc640 10 monclient: build_initial_monmap 2024-03-25T17:44:44.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.452+0000 7fae18dbc640 10 monclient: monmap: 2024-03-25T17:44:44.453 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:44.453 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:44.453 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:44:44.453853+0000 2024-03-25T17:44:44.453 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:44:44.453853+0000 2024-03-25T17:44:44.454 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:44.454 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:44.454 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:44.454 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:44:44.454 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:44:44.454 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:44.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.453+0000 7fae18dbc640 1 Processor -- start 2024-03-25T17:44:44.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.453+0000 7fae18dbc640 1 -- start start 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.453+0000 7fae18dbc640 10 monclient: init 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fae1406ab80 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 10 monclient(hunting): picked mon.noname-a con 0x7fae1406af60 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fae140b00b0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae140b0500 con 0x7fae1406af60 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1413ffa0 con 0x7fae1406ab80 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae140af520 con 0x7fae140b00b0 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:44.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.454+0000 7fae18dbc640 10 monclient(hunting): authenticate will time out at 6287.682500s 2024-03-25T17:44:44.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.456+0000 7fae12d9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fae140b00b0 0x7fae1414dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:38300/0 (socket says 172.21.15.44:38300) 2024-03-25T17:44:44.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.456+0000 7fae12d9d640 1 -- 172.21.15.44:0/864156422 learned_addr learned my addr 172.21.15.44:0/864156422 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:44.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.463+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695247228 0 0) 0x7fae140af520 con 0x7fae140b00b0 2024-03-25T17:44:44.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.463+0000 7fae1159a640 10 monclient(hunting): _negotiate 2024-03-25T17:44:44.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.463+0000 7fae1159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:44.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.463+0000 7fae1159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.463+0000 7fae1159a640 10 monclient(hunting): my global_id is 4472 2024-03-25T17:44:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.463+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc003520 con 0x7fae140b00b0 2024-03-25T17:44:44.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1754299153 0 0) 0x7fadfc003520 con 0x7fae140b00b0 2024-03-25T17:44:44.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 >> v1:172.21.15.111:6789/0 conn(0x7fae1406ab80 legacy=0x7fae140ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:44:44.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 >> v1:172.21.15.44:6789/0 conn(0x7fae1406af60 legacy=0x7fae1413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:44.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 10 monclient: _finish_hunting 0 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 1 monclient: found mon.noname-b 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae14072890 con 0x7fae140b00b0 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 10 monclient: _finish_auth 0 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 10 monclient: _check_auth_tickets 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.468+0000 7fae1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.469+0000 7fae18dbc640 5 monclient: authenticate success, global_id 4472 2024-03-25T17:44:44.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.469+0000 7fae18dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:44.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fae0c001a10 con 0x7fae140b00b0 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 10 monclient: dump: 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:44.478 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae0c001ce0 con 0x7fae140b00b0 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 1 -- 172.21.15.44:0/864156422 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fae0c002070 con 0x7fae140b00b0 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae18dbc640 10 monclient: get_monmap_and_config success 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae18dbc640 4 set_mon_vals no callback set 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.477+0000 7fae1159a640 10 monclient: dump: 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:44.479 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:44.480 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:44.480 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:44.480 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:44.480 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:44.480 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:44.480 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:44.480 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.478+0000 7fae18dbc640 10 monclient: shutdown 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.478+0000 7fae18dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.478+0000 7fae18dbc640 1 -- 172.21.15.44:0/864156422 >> v1:172.21.15.44:6790/0 conn(0x7fae140b00b0 legacy=0x7fae1414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.478+0000 7fae18dbc640 1 -- 172.21.15.44:0/864156422 shutdown_connections 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.478+0000 7fae18dbc640 1 -- 172.21.15.44:0/864156422 >> 172.21.15.44:0/864156422 conn(0x7fae140b4a80 msgr2=0x7fae140b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.478+0000 7fae18dbc640 1 -- 172.21.15.44:0/864156422 shutdown_connections 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.478+0000 7fae18dbc640 1 -- 172.21.15.44:0/864156422 wait complete. 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 10 monclient: build_initial_monmap 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 10 monclient: monmap: 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 1 librados: starting msgr at 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 1 librados: starting objecter 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 1 Processor -- start 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 1 -- start start 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 1 librados: setting wanted keys 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 1 librados: calling monclient init 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.479+0000 7fae18dbc640 10 monclient: init 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:44:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 10 monclient(hunting): picked mon.noname-a con 0x7fae1406ab80 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fae140b00b0 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fae14144070 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae14145020 con 0x7fae1406ab80 2024-03-25T17:44:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae14145300 con 0x7fae140b00b0 2024-03-25T17:44:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae14145ba0 con 0x7fae14144070 2024-03-25T17:44:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.480+0000 7fae18dbc640 10 monclient(hunting): authenticate will time out at 6287.708302s 2024-03-25T17:44:44.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.490+0000 7fae12d9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fae14144070 0x7fae141d4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:38324/0 (socket says 172.21.15.44:38324) 2024-03-25T17:44:44.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.490+0000 7fae12d9d640 1 -- 172.21.15.44:0/1872110103 learned_addr learned my addr 172.21.15.44:0/1872110103 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:44.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.491+0000 7fae02ffd640 10 client.?.objecter ms_handle_connect 0x7fae14144070 2024-03-25T17:44:44.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.491+0000 7fae02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:44.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.492+0000 7fae02ffd640 10 client.?.objecter ms_handle_connect 0x7fae140b00b0 2024-03-25T17:44:44.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.492+0000 7fae02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.499+0000 7fae02ffd640 10 client.?.objecter ms_handle_connect 0x7fae1406ab80 2024-03-25T17:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.499+0000 7fae02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.500+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42858623 0 0) 0x7fae14145300 con 0x7fae140b00b0 2024-03-25T17:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.500+0000 7fae02ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.500+0000 7fae02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.500+0000 7fae02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.500+0000 7fae02ffd640 10 monclient(hunting): my global_id is 4471 2024-03-25T17:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.500+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf4004eb0 con 0x7fae140b00b0 2024-03-25T17:44:44.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.506+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4252996049 0 0) 0x7fadf4004eb0 con 0x7fae140b00b0 2024-03-25T17:44:44.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.506+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae14145300 con 0x7fae140b00b0 2024-03-25T17:44:44.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.506+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fae08001930 con 0x7fae140b00b0 2024-03-25T17:44:44.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.506+0000 7fae02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:44.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.507+0000 7fae02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:44:44.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.507+0000 7fae02ffd640 10 monclient(hunting): dump: 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:44.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.507+0000 7fae02ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:44:44.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.511+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846238241 0 0) 0x7fae14145ba0 con 0x7fae14144070 2024-03-25T17:44:44.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.511+0000 7fae02ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:44:44.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.511+0000 7fae02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:44.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.511+0000 7fae02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:44.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.511+0000 7fae02ffd640 10 monclient(hunting): my global_id is 4475 2024-03-25T17:44:44.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.511+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf4004eb0 con 0x7fae14144070 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 912599359 0 0) 0x7fae14145300 con 0x7fae140b00b0 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 >> v1:172.21.15.44:6790/0 conn(0x7fae14144070 legacy=0x7fae141d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 >> v1:172.21.15.44:6789/0 conn(0x7fae1406ab80 legacy=0x7fae14142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 1 monclient: found mon.b 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae141d4f20 con 0x7fae140b00b0 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 10 monclient: _finish_auth 0 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:44:44.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 5 monclient: authenticate success, global_id 4471 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 10 monclient: _renew_subs 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae141d51d0 con 0x7fae140b00b0 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 10 monclient: _renew_subs 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae141dad50 con 0x7fae140b00b0 2024-03-25T17:44:44.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.514+0000 7fae18dbc640 1 librados: init done 2024-03-25T17:44:44.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.515+0000 7fae18dbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:44.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.515+0000 7fae18dbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:44.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.515+0000 7fae18dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:44.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.515+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae14145a00 con 0x7fae140b00b0 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.802+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae08003610 con 0x7fae140b00b0 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.802+0000 7fae02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.802+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fae08003840 con 0x7fae140b00b0 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.802+0000 7fae02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.802+0000 7fae02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.802+0000 7fae02ffd640 10 monclient: dump: 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:44.803 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:44.804 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:44.804 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:44.804 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:44.804 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:44.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.802+0000 7fae10d99640 4 set_mon_vals no callback set 2024-03-25T17:44:44.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.803+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fae08003a60 con 0x7fae140b00b0 2024-03-25T17:44:44.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.804+0000 7fae1259c640 10 monclient: get_auth_request con 0x7fadf406dd90 auth_method 0 2024-03-25T17:44:44.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.804+0000 7fae02ffd640 10 client.4471.objecter ms_handle_connect 0x7fadf406dd90 2024-03-25T17:44:44.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.809+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fae08053420 con 0x7fae140b00b0 2024-03-25T17:44:44.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.809+0000 7fae02ffd640 10 client.4471.objecter ms_dispatch 0x7fae14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:44:44.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.809+0000 7fae02ffd640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:44:44.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.809+0000 7fae02ffd640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:44:44.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.809+0000 7fae02ffd640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-25T17:44:44.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.809+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae0804ef30 con 0x7fae140b00b0 2024-03-25T17:44:44.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.809+0000 7fae02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:44.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:44.809+0000 7fae02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:44:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.061+0000 7fae18dbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.061+0000 7fae18dbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.061+0000 7fae18dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:44:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.061+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae14145d30 con 0x7fae140b00b0 2024-03-25T17:44:45.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.087+0000 7fae02ffd640 1 -- 172.21.15.44:0/1872110103 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fae08050e60 con 0x7fae140b00b0 2024-03-25T17:44:45.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.087+0000 7fae02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:45.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.087+0000 7fae02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:44:45.088 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:44:45.088 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 10 librados: watch_flush enter 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 10 librados: watch_flush exit 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 10 client.4471.objecter successfully canceled tick 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 >> v1:172.21.15.44:6800/3257043132 conn(0x7fadf406dd90 legacy=0x7fadf4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 10 monclient: shutdown 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 >> v1:172.21.15.111:6789/0 conn(0x7fae140b00b0 legacy=0x7fae141460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 shutdown_connections 2024-03-25T17:44:45.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 >> 172.21.15.44:0/1872110103 conn(0x7fae140b4a80 msgr2=0x7fae140ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:45.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 shutdown_connections 2024-03-25T17:44:45.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 1 -- 172.21.15.44:0/1872110103 wait complete. 2024-03-25T17:44:45.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:45.090+0000 7fae18dbc640 1 librados: shutdown 2024-03-25T17:44:45.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:44:51.909 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:44:52.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.024+0000 7f908e139640 10 monclient: get_monmap_and_config 2024-03-25T17:44:52.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.024+0000 7f908e139640 10 monclient: build_initial_monmap 2024-03-25T17:44:52.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.024+0000 7f908e139640 10 monclient: monmap: 2024-03-25T17:44:52.024 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:52.024 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:52.025 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:44:52.025042+0000 2024-03-25T17:44:52.025 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:44:52.025042+0000 2024-03-25T17:44:52.025 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:52.025 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:52.025 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:52.025 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:44:52.025 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:44:52.025 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.024+0000 7f908e139640 1 Processor -- start 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.025+0000 7f908e139640 1 -- start start 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.025+0000 7f908e139640 10 monclient: init 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.025+0000 7f908e139640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.025+0000 7f908e139640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.025+0000 7f908e139640 10 monclient(hunting): picked mon.noname-b con 0x7f908806c380 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.026+0000 7f908e139640 10 monclient(hunting): picked mon.noname-c con 0x7f908806d670 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.026+0000 7f908e139640 10 monclient(hunting): picked mon.noname-a con 0x7f90880af8f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.026+0000 7f908e139640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908806c9a0 con 0x7f90880af8f0 2024-03-25T17:44:52.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.026+0000 7f908e139640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90880ada90 con 0x7f908806d670 2024-03-25T17:44:52.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.026+0000 7f908e139640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90880b0d10 con 0x7f908806c380 2024-03-25T17:44:52.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.026+0000 7f908e139640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:52.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.026+0000 7f908e139640 10 monclient(hunting): authenticate will time out at 6295.253736s 2024-03-25T17:44:52.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.031+0000 7f9086ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f908806d670 0x7f90880ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45868/0 (socket says 172.21.15.44:45868) 2024-03-25T17:44:52.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.031+0000 7f9086ffd640 1 -- 172.21.15.44:0/1432707985 learned_addr learned my addr 172.21.15.44:0/1432707985 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:52.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.060+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092603795 0 0) 0x7f908806c9a0 con 0x7f90880af8f0 2024-03-25T17:44:52.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.060+0000 7f90867fc640 10 monclient(hunting): _negotiate 2024-03-25T17:44:52.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.060+0000 7f90867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:52.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.060+0000 7f90867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:52.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.060+0000 7f90867fc640 10 monclient(hunting): my global_id is 4491 2024-03-25T17:44:52.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.060+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9070003520 con 0x7f90880af8f0 2024-03-25T17:44:52.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604149589 0 0) 0x7f9070003520 con 0x7f90880af8f0 2024-03-25T17:44:52.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 >> v1:172.21.15.44:6790/0 conn(0x7f908806c380 legacy=0x7f90880b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:52.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 >> v1:172.21.15.111:6789/0 conn(0x7f908806d670 legacy=0x7f90880ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:52.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 10 monclient: _finish_hunting 0 2024-03-25T17:44:52.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 1 monclient: found mon.noname-a 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90880add00 con 0x7f90880af8f0 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 10 monclient: _finish_auth 0 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 10 monclient: _check_auth_tickets 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9080001a10 con 0x7f90880af8f0 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f90867fc640 10 monclient: dump: 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.134 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:52.135 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:52.135 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:52.135 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:52.135 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:52.135 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:52.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f908e139640 5 monclient: authenticate success, global_id 4491 2024-03-25T17:44:52.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.132+0000 7f908e139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9080001b70 con 0x7f90880af8f0 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f90867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f90867fc640 1 -- 172.21.15.44:0/1432707985 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9080001f00 con 0x7f90880af8f0 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f90867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f90867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f90867fc640 10 monclient: dump: 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.145 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:52.146 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:52.146 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:52.146 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:52.146 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:52.146 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:52.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 10 monclient: get_monmap_and_config success 2024-03-25T17:44:52.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 4 set_mon_vals no callback set 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 10 monclient: shutdown 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 1 -- 172.21.15.44:0/1432707985 >> v1:172.21.15.44:6789/0 conn(0x7f90880af8f0 legacy=0x7f90880aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 1 -- 172.21.15.44:0/1432707985 shutdown_connections 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 1 -- 172.21.15.44:0/1432707985 >> 172.21.15.44:0/1432707985 conn(0x7f90880b5c00 msgr2=0x7f90880b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 1 -- 172.21.15.44:0/1432707985 shutdown_connections 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.145+0000 7f908e139640 1 -- 172.21.15.44:0/1432707985 wait complete. 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.146+0000 7f908e139640 10 monclient: build_initial_monmap 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.146+0000 7f908e139640 10 monclient: monmap: 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:44:52.148 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.146+0000 7f908e139640 1 librados: starting msgr at 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.146+0000 7f908e139640 1 librados: starting objecter 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.146+0000 7f908e139640 1 Processor -- start 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 1 -- start start 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 1 librados: setting wanted keys 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 1 librados: calling monclient init 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 10 monclient: init 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:44:52.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 10 monclient(hunting): picked mon.noname-c con 0x7f908806c380 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:52.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 10 monclient(hunting): picked mon.noname-a con 0x7f908806d670 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:52.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 10 monclient(hunting): picked mon.noname-b con 0x7f90880af8f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:52.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9088141280 con 0x7f908806d670 2024-03-25T17:44:52.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90881495e0 con 0x7f90880af8f0 2024-03-25T17:44:52.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908813eb30 con 0x7f908806c380 2024-03-25T17:44:52.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:52.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.147+0000 7f908e139640 10 monclient(hunting): authenticate will time out at 6295.375404s 2024-03-25T17:44:52.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.163+0000 7f9086ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f908806d670 0x7f908813dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:48732/0 (socket says 172.21.15.44:48732) 2024-03-25T17:44:52.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.163+0000 7f9086ffd640 1 -- 172.21.15.44:0/83423992 learned_addr learned my addr 172.21.15.44:0/83423992 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:52.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.166+0000 7f906ffff640 10 client.?.objecter ms_handle_connect 0x7f908806d670 2024-03-25T17:44:52.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.166+0000 7f906ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606796936 0 0) 0x7f9088141280 con 0x7f908806d670 2024-03-25T17:44:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:44:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 10 monclient(hunting): my global_id is 4494 2024-03-25T17:44:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9074004eb0 con 0x7f908806d670 2024-03-25T17:44:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 10 client.?.objecter ms_handle_connect 0x7f90880af8f0 2024-03-25T17:44:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 10 client.?.objecter ms_handle_connect 0x7f908806c380 2024-03-25T17:44:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.185+0000 7f906ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:52.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.201+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750038770 0 0) 0x7f908813eb30 con 0x7f908806c380 2024-03-25T17:44:52.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.201+0000 7f906ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:44:52.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.201+0000 7f906ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:52.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.201+0000 7f906ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:52.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.201+0000 7f906ffff640 10 monclient(hunting): my global_id is 4481 2024-03-25T17:44:52.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.201+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9088141280 con 0x7f908806c380 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.204+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4133650432 0 0) 0x7f9074004eb0 con 0x7f908806d670 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.204+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f908813eb30 con 0x7f908806d670 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.204+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f907c001930 con 0x7f908806d670 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.204+0000 7f906ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.204+0000 7f906ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.204+0000 7f906ffff640 10 monclient(hunting): dump: 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.205 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:52.206 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:52.206 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:52.206 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:52.206 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:52.206 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:52.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.204+0000 7f906ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:44:52.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.218+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169654109 0 0) 0x7f90881495e0 con 0x7f90880af8f0 2024-03-25T17:44:52.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.218+0000 7f906ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:44:52.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.218+0000 7f906ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:52.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.218+0000 7f906ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:52.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.218+0000 7f906ffff640 10 monclient(hunting): my global_id is 4477 2024-03-25T17:44:52.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.218+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9074004eb0 con 0x7f90880af8f0 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.230+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1146340166 0 0) 0x7f9088141280 con 0x7f908806c380 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.230+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90881495e0 con 0x7f908806c380 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.230+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9078001820 con 0x7f908806c380 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.230+0000 7f906ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.230+0000 7f906ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.230+0000 7f906ffff640 10 monclient(hunting): dump: 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.231 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:52.232 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:52.232 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:52.232 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:52.232 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:52.232 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2974380579 0 0) 0x7f908813eb30 con 0x7f908806d670 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 >> v1:172.21.15.44:6790/0 conn(0x7f908806c380 legacy=0x7f90881da760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 >> v1:172.21.15.111:6789/0 conn(0x7f90880af8f0 legacy=0x7f908813e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 1 monclient: found mon.a 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088140200 con 0x7f908806d670 2024-03-25T17:44:52.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 10 monclient: _finish_auth 0 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f906ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 5 monclient: authenticate success, global_id 4494 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 10 monclient: _renew_subs 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90881403f0 con 0x7f908806d670 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 10 monclient: _renew_subs 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9088140dd0 con 0x7f908806d670 2024-03-25T17:44:52.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.237+0000 7f908e139640 1 librados: init done 2024-03-25T17:44:52.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.239+0000 7f908e139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:52.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.239+0000 7f908e139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:52.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.239+0000 7f908e139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:52.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.239+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9088141430 con 0x7f908806d670 2024-03-25T17:44:52.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.548+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f907c0047e0 con 0x7f908806d670 2024-03-25T17:44:52.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.548+0000 7f906ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.548+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f907c004a10 con 0x7f908806d670 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.548+0000 7f906ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.548+0000 7f906ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.548+0000 7f906ffff640 10 monclient: dump: 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:52.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.548+0000 7f9085ffb640 4 set_mon_vals no callback set 2024-03-25T17:44:52.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.549+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f907c004c30 con 0x7f908806d670 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.549+0000 7f90877fe640 10 monclient: get_auth_request con 0x7f907406dfb0 auth_method 0 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.549+0000 7f906ffff640 10 client.4494.objecter ms_handle_connect 0x7f907406dfb0 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.555+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f907c054670 con 0x7f908806d670 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.555+0000 7f906ffff640 10 client.4494.objecter ms_dispatch 0x7f9088000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.555+0000 7f906ffff640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.555+0000 7f906ffff640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.555+0000 7f906ffff640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.555+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f907c0b18b0 con 0x7f908806d670 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.555+0000 7f906ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:52.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.555+0000 7f906ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:44:52.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.819+0000 7f908e139640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:52.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.819+0000 7f908e139640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.819+0000 7f908e139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.819+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90880627b0 con 0x7f908806d670 2024-03-25T17:44:52.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.863+0000 7f906ffff640 1 -- 172.21.15.44:0/83423992 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f907c050180 con 0x7f908806d670 2024-03-25T17:44:52.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.863+0000 7f906ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:44:52.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.863+0000 7f906ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:44:52.864 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:44:52.864 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:44:52.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 10 librados: watch_flush enter 2024-03-25T17:44:52.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 10 librados: watch_flush exit 2024-03-25T17:44:52.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-25T17:44:52.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 10 client.4494.objecter successfully canceled tick 2024-03-25T17:44:52.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 >> v1:172.21.15.44:6800/3257043132 conn(0x7f907406dfb0 legacy=0x7f9074070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:52.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 10 monclient: shutdown 2024-03-25T17:44:52.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:52.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 >> v1:172.21.15.44:6789/0 conn(0x7f908806d670 legacy=0x7f908813dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:52.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 shutdown_connections 2024-03-25T17:44:52.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.865+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 >> 172.21.15.44:0/83423992 conn(0x7f90880b5c00 msgr2=0x7f9088148590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:52.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.866+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 shutdown_connections 2024-03-25T17:44:52.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.866+0000 7f908e139640 1 -- 172.21.15.44:0/83423992 wait complete. 2024-03-25T17:44:52.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:52.866+0000 7f908e139640 1 librados: shutdown 2024-03-25T17:44:52.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:44:58.132 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:15.487 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:58.138 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:15.494 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:58.186 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:15.542 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:58.186 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:15.542 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:58.191 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:15.511 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:58.196 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:15.516 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:58.199 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:15.555 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:58.221 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:15.541 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:58.223 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:15.579 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:44:59.684 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:44:59.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.797+0000 7f70578c0640 10 monclient: get_monmap_and_config 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.797+0000 7f70578c0640 10 monclient: build_initial_monmap 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.797+0000 7f70578c0640 10 monclient: monmap: 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:44:59.799398+0000 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:44:59.799398+0000 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:44:59.799 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:59.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.798+0000 7f70578c0640 1 Processor -- start 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.798+0000 7f70578c0640 1 -- start start 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.798+0000 7f70578c0640 10 monclient: init 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 10 monclient(hunting): picked mon.noname-a con 0x7f70500a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 10 monclient(hunting): picked mon.noname-c con 0x7f705013f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 10 monclient(hunting): picked mon.noname-b con 0x7f70501430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7050142520 con 0x7f70500a98f0 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70500b28a0 con 0x7f705013f710 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70500b2a30 con 0x7f70501430c0 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:59.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.799+0000 7f70578c0640 10 monclient(hunting): authenticate will time out at 6303.028046s 2024-03-25T17:44:59.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.813+0000 7f705565c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f70500a98f0 0x7f70500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56078/0 (socket says 172.21.15.44:56078) 2024-03-25T17:44:59.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.813+0000 7f705565c640 1 -- 172.21.15.44:0/3769531399 learned_addr learned my addr 172.21.15.44:0/3769531399 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:59.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.826+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128267459 0 0) 0x7f7050142520 con 0x7f70500a98f0 2024-03-25T17:44:59.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.826+0000 7f7047fff640 10 monclient(hunting): _negotiate 2024-03-25T17:44:59.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.826+0000 7f7047fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.826+0000 7f7047fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.826+0000 7f7047fff640 10 monclient(hunting): my global_id is 4497 2024-03-25T17:44:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.826+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f703c003520 con 0x7f70500a98f0 2024-03-25T17:44:59.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.833+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961116386 0 0) 0x7f70500b28a0 con 0x7f705013f710 2024-03-25T17:44:59.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.833+0000 7f7047fff640 10 monclient(hunting): _negotiate 2024-03-25T17:44:59.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.833+0000 7f7047fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:59.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.833+0000 7f7047fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:59.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.833+0000 7f7047fff640 10 monclient(hunting): my global_id is 4480 2024-03-25T17:44:59.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.833+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7050142520 con 0x7f705013f710 2024-03-25T17:44:59.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.844+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728371195 0 0) 0x7f703c003520 con 0x7f70500a98f0 2024-03-25T17:44:59.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.844+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 >> v1:172.21.15.44:6790/0 conn(0x7f70501430c0 legacy=0x7f705014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:59.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.844+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 >> v1:172.21.15.111:6789/0 conn(0x7f705013f710 legacy=0x7f70501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.844+0000 7f7047fff640 10 monclient: _finish_hunting 0 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.844+0000 7f7047fff640 1 monclient: found mon.noname-a 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70500b44c0 con 0x7f70500a98f0 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 10 monclient: _finish_auth 0 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 10 monclient: _check_auth_tickets 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f704c001690 con 0x7f70500a98f0 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f70578c0640 5 monclient: authenticate success, global_id 4497 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f70578c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f7047fff640 10 monclient: dump: 2024-03-25T17:44:59.847 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:59.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.845+0000 7f70578c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:44:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.878+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f704c0017f0 con 0x7f70500a98f0 2024-03-25T17:44:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.878+0000 7f7047fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:44:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.878+0000 7f7047fff640 1 -- 172.21.15.44:0/3769531399 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f704c001c00 con 0x7f70500a98f0 2024-03-25T17:44:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.878+0000 7f70578c0640 10 monclient: get_monmap_and_config success 2024-03-25T17:44:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.878+0000 7f70578c0640 4 set_mon_vals no callback set 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.878+0000 7f7047fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.878+0000 7f7047fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.878+0000 7f7047fff640 10 monclient: dump: 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:59.881 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:59.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.879+0000 7f70578c0640 10 monclient: shutdown 2024-03-25T17:44:59.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.879+0000 7f70578c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:44:59.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.879+0000 7f70578c0640 1 -- 172.21.15.44:0/3769531399 >> v1:172.21.15.44:6789/0 conn(0x7f70500a98f0 legacy=0x7f70500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.879+0000 7f70578c0640 1 -- 172.21.15.44:0/3769531399 shutdown_connections 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.879+0000 7f70578c0640 1 -- 172.21.15.44:0/3769531399 >> 172.21.15.44:0/3769531399 conn(0x7f70500b7ea0 msgr2=0x7f70500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.879+0000 7f70578c0640 1 -- 172.21.15.44:0/3769531399 shutdown_connections 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.879+0000 7f70578c0640 1 -- 172.21.15.44:0/3769531399 wait complete. 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.880+0000 7f70578c0640 10 monclient: build_initial_monmap 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.880+0000 7f70578c0640 10 monclient: monmap: 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:44:59.884 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.880+0000 7f70578c0640 1 librados: starting msgr at 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.880+0000 7f70578c0640 1 librados: starting objecter 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.880+0000 7f70578c0640 1 Processor -- start 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 1 -- start start 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 1 librados: setting wanted keys 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 1 librados: calling monclient init 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 10 monclient: init 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 10 monclient: _reopen_session rank -1 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 10 monclient(hunting): picked mon.noname-a con 0x7f70500a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 10 monclient(hunting): picked mon.noname-b con 0x7f705013f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:44:59.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 10 monclient(hunting): picked mon.noname-c con 0x7f70501430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:44:59.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70500b7ce0 con 0x7f70500a98f0 2024-03-25T17:44:59.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70501319b0 con 0x7f705013f710 2024-03-25T17:44:59.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70500b7800 con 0x7f70501430c0 2024-03-25T17:44:59.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 10 monclient(hunting): _renew_subs 2024-03-25T17:44:59.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.881+0000 7f70578c0640 10 monclient(hunting): authenticate will time out at 6303.110345s 2024-03-25T17:44:59.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.894+0000 7f705565c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f70500a98f0 0x7f70501dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56144/0 (socket says 172.21.15.44:56144) 2024-03-25T17:44:59.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.894+0000 7f705565c640 1 -- 172.21.15.44:0/2621748487 learned_addr learned my addr 172.21.15.44:0/2621748487 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:44:59.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.895+0000 7f7045ffb640 10 client.?.objecter ms_handle_connect 0x7f705013f710 2024-03-25T17:44:59.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.895+0000 7f7045ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:59.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.897+0000 7f7045ffb640 10 client.?.objecter ms_handle_connect 0x7f70500a98f0 2024-03-25T17:44:59.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.897+0000 7f7045ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:59.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.900+0000 7f7045ffb640 10 client.?.objecter ms_handle_connect 0x7f70501430c0 2024-03-25T17:44:59.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.900+0000 7f7045ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:44:59.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.909+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206884523 0 0) 0x7f70500b7ce0 con 0x7f70500a98f0 2024-03-25T17:44:59.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.909+0000 7f7045ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:44:59.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.909+0000 7f7045ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:59.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.909+0000 7f7045ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:59.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.909+0000 7f7045ffb640 10 monclient(hunting): my global_id is 4500 2024-03-25T17:44:59.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.909+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7034004eb0 con 0x7f70500a98f0 2024-03-25T17:44:59.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.911+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242887265 0 0) 0x7f70501319b0 con 0x7f705013f710 2024-03-25T17:44:59.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.911+0000 7f7045ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:44:59.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.911+0000 7f7045ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:59.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.911+0000 7f7045ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:59.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.911+0000 7f7045ffb640 10 monclient(hunting): my global_id is 4483 2024-03-25T17:44:59.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.911+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70500b7ce0 con 0x7f705013f710 2024-03-25T17:44:59.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.913+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854648361 0 0) 0x7f70500b7800 con 0x7f70501430c0 2024-03-25T17:44:59.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.913+0000 7f7045ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:44:59.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.913+0000 7f7045ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:44:59.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.913+0000 7f7045ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:44:59.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.913+0000 7f7045ffb640 10 monclient(hunting): my global_id is 4487 2024-03-25T17:44:59.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.913+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70501319b0 con 0x7f70501430c0 2024-03-25T17:44:59.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2914934131 0 0) 0x7f70500b7ce0 con 0x7f705013f710 2024-03-25T17:44:59.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70500b7800 con 0x7f705013f710 2024-03-25T17:44:59.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7040001930 con 0x7f705013f710 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 10 monclient(hunting): dump: 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:59.926 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647029494 0 0) 0x7f7034004eb0 con 0x7f70500a98f0 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70500b7ce0 con 0x7f70500a98f0 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f704c001c40 con 0x7f70500a98f0 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.923+0000 7f7045ffb640 10 monclient(hunting): dump: 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:59.927 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:59.929 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:59.929 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:59.929 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:59.929 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:59.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.934+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2054704648 0 0) 0x7f70501319b0 con 0x7f70501430c0 2024-03-25T17:44:59.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.934+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7034004eb0 con 0x7f70501430c0 2024-03-25T17:44:59.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.934+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7048001a60 con 0x7f70501430c0 2024-03-25T17:44:59.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.934+0000 7f7045ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:44:59.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.934+0000 7f7045ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:44:59.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.934+0000 7f7045ffb640 10 monclient(hunting): dump: 2024-03-25T17:44:59.937 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:44:59.937 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:44:59.937 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.937 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:44:59.937 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:44:59.937 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:44:59.938 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:44:59.938 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:44:59.938 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:44:59.938 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:44:59.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3448236436 0 0) 0x7f70500b7ce0 con 0x7f70500a98f0 2024-03-25T17:44:59.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 >> v1:172.21.15.44:6790/0 conn(0x7f70501430c0 legacy=0x7f70501312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:59.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 >> v1:172.21.15.111:6789/0 conn(0x7f705013f710 legacy=0x7f7050130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:44:59.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:44:59.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 1 monclient: found mon.a 2024-03-25T17:44:59.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:44:59.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:59.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7050136f50 con 0x7f70500a98f0 2024-03-25T17:44:59.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 10 monclient: _finish_auth 0 2024-03-25T17:44:59.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:44:59.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f7045ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:44:59.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 5 monclient: authenticate success, global_id 4500 2024-03-25T17:44:59.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 10 monclient: _renew_subs 2024-03-25T17:44:59.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:59.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70501350f0 con 0x7f70500a98f0 2024-03-25T17:44:59.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:44:59.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 10 monclient: _renew_subs 2024-03-25T17:44:59.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:59.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7050135c60 con 0x7f70500a98f0 2024-03-25T17:44:59.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.944+0000 7f70578c0640 1 librados: init done 2024-03-25T17:44:59.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.945+0000 7f70578c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:59.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.945+0000 7f70578c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:44:59.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.945+0000 7f70578c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:44:59.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:44:59.945+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70500627b0 con 0x7f70500a98f0 2024-03-25T17:45:00.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.269+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f704c004a90 con 0x7f70500a98f0 2024-03-25T17:45:00.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.269+0000 7f7045ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.269+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f704c004de0 con 0x7f70500a98f0 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.269+0000 7f7045ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.269+0000 7f7045ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.269+0000 7f7045ffb640 10 monclient: dump: 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:00.272 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:00.273 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:00.273 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:00.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.269+0000 7f70477fe640 4 set_mon_vals no callback set 2024-03-25T17:45:00.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.271+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f704c005000 con 0x7f70500a98f0 2024-03-25T17:45:00.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.271+0000 7f7054e5b640 10 monclient: get_auth_request con 0x7f703406e160 auth_method 0 2024-03-25T17:45:00.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.271+0000 7f7045ffb640 10 client.4500.objecter ms_handle_connect 0x7f703406e160 2024-03-25T17:45:00.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.286+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f704c054970 con 0x7f70500a98f0 2024-03-25T17:45:00.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.286+0000 7f7045ffb640 10 client.4500.objecter ms_dispatch 0x7f7050000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:45:00.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.286+0000 7f7045ffb640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:45:00.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.286+0000 7f7045ffb640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:45:00.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.286+0000 7f7045ffb640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-25T17:45:00.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.286+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f704c050480 con 0x7f70500a98f0 2024-03-25T17:45:00.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.286+0000 7f7045ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:00.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.286+0000 7f7045ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:45:00.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.509+0000 7f70578c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:00.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.509+0000 7f70578c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:00.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.509+0000 7f70578c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:00.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.509+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f705013e270 con 0x7f70500a98f0 2024-03-25T17:45:00.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.550+0000 7f7045ffb640 1 -- 172.21.15.44:0/2621748487 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f704c0523b0 con 0x7f70500a98f0 2024-03-25T17:45:00.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.550+0000 7f7045ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:00.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.550+0000 7f7045ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:45:00.552 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:45:00.552 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:45:00.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 10 librados: watch_flush enter 2024-03-25T17:45:00.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 10 librados: watch_flush exit 2024-03-25T17:45:00.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-25T17:45:00.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 10 client.4500.objecter successfully canceled tick 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 >> v1:172.21.15.44:6800/3257043132 conn(0x7f703406e160 legacy=0x7f7034070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 10 monclient: shutdown 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 >> v1:172.21.15.44:6789/0 conn(0x7f70500a98f0 legacy=0x7f70501dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 shutdown_connections 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 >> 172.21.15.44:0/2621748487 conn(0x7f70500b7ea0 msgr2=0x7f7050074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 shutdown_connections 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.552+0000 7f70578c0640 1 -- 172.21.15.44:0/2621748487 wait complete. 2024-03-25T17:45:00.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:00.553+0000 7f70578c0640 1 librados: shutdown 2024-03-25T17:45:00.564 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:45:07.371 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:45:07.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.484+0000 7fd7ab30e640 10 monclient: get_monmap_and_config 2024-03-25T17:45:07.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.484+0000 7fd7ab30e640 10 monclient: build_initial_monmap 2024-03-25T17:45:07.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.484+0000 7fd7ab30e640 10 monclient: monmap: 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:45:07.485265+0000 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:45:07.485265+0000 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:45:07.486 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:07.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.485+0000 7fd7ab30e640 1 Processor -- start 2024-03-25T17:45:07.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.485+0000 7fd7ab30e640 1 -- start start 2024-03-25T17:45:07.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.485+0000 7fd7ab30e640 10 monclient: init 2024-03-25T17:45:07.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:07.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:45:07.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a40a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:07.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a413f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:07.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a41430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:07.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a4142520 con 0x7fd7a41430c0 2024-03-25T17:45:07.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40b28a0 con 0x7fd7a40a98f0 2024-03-25T17:45:07.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40b2a30 con 0x7fd7a413f710 2024-03-25T17:45:07.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:07.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.486+0000 7fd7ab30e640 10 monclient(hunting): authenticate will time out at 6310.713871s 2024-03-25T17:45:07.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.489+0000 7fd7a98ab640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd7a41430c0 0x7fd7a414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:60804/0 (socket says 172.21.15.44:60804) 2024-03-25T17:45:07.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.489+0000 7fd7a98ab640 1 -- 172.21.15.44:0/4126467140 learned_addr learned my addr 172.21.15.44:0/4126467140 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:07.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.505+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782096340 0 0) 0x7fd7a4142520 con 0x7fd7a41430c0 2024-03-25T17:45:07.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.505+0000 7fd79bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:07.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.505+0000 7fd79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:07.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.505+0000 7fd79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:07.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.505+0000 7fd79bfff640 10 monclient(hunting): my global_id is 4503 2024-03-25T17:45:07.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.505+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd790003520 con 0x7fd7a41430c0 2024-03-25T17:45:07.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.508+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112098075 0 0) 0x7fd7a40b28a0 con 0x7fd7a40a98f0 2024-03-25T17:45:07.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.508+0000 7fd79bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:07.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.508+0000 7fd79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:07.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.508+0000 7fd79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:07.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.508+0000 7fd79bfff640 10 monclient(hunting): my global_id is 4486 2024-03-25T17:45:07.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.508+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a4142520 con 0x7fd7a40a98f0 2024-03-25T17:45:07.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.512+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495065712 0 0) 0x7fd7a40b2a30 con 0x7fd7a413f710 2024-03-25T17:45:07.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.512+0000 7fd79bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:07.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.512+0000 7fd79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:07.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.512+0000 7fd79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:07.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.512+0000 7fd79bfff640 10 monclient(hunting): my global_id is 4490 2024-03-25T17:45:07.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.512+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a40b28a0 con 0x7fd7a413f710 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787661435 0 0) 0x7fd790003520 con 0x7fd7a41430c0 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 >> v1:172.21.15.44:6790/0 conn(0x7fd7a413f710 legacy=0x7fd7a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 >> v1:172.21.15.111:6789/0 conn(0x7fd7a40a98f0 legacy=0x7fd7a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 1 monclient: found mon.noname-a 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a40b44c0 con 0x7fd7a41430c0 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 10 monclient: _finish_auth 0 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:45:07.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd79c001a10 con 0x7fd7a41430c0 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd79bfff640 10 monclient: dump: 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:07.537 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:07.538 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:07.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd7ab30e640 5 monclient: authenticate success, global_id 4503 2024-03-25T17:45:07.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.535+0000 7fd7ab30e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:07.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.557+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd79c001b70 con 0x7fd7a41430c0 2024-03-25T17:45:07.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.557+0000 7fd79bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:07.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.557+0000 7fd79bfff640 1 -- 172.21.15.44:0/4126467140 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd79c001f80 con 0x7fd7a41430c0 2024-03-25T17:45:07.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.557+0000 7fd7ab30e640 10 monclient: get_monmap_and_config success 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.557+0000 7fd7ab30e640 4 set_mon_vals no callback set 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.557+0000 7fd79bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.557+0000 7fd79bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.557+0000 7fd79bfff640 10 monclient: dump: 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:07.558 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:07.559 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:07.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.558+0000 7fd7ab30e640 10 monclient: shutdown 2024-03-25T17:45:07.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.558+0000 7fd7ab30e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:07.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.558+0000 7fd7ab30e640 1 -- 172.21.15.44:0/4126467140 >> v1:172.21.15.44:6789/0 conn(0x7fd7a41430c0 legacy=0x7fd7a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.558+0000 7fd7ab30e640 1 -- 172.21.15.44:0/4126467140 shutdown_connections 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.558+0000 7fd7ab30e640 1 -- 172.21.15.44:0/4126467140 >> 172.21.15.44:0/4126467140 conn(0x7fd7a40b7ea0 msgr2=0x7fd7a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.558+0000 7fd7ab30e640 1 -- 172.21.15.44:0/4126467140 shutdown_connections 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.558+0000 7fd7ab30e640 1 -- 172.21.15.44:0/4126467140 wait complete. 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 10 monclient: build_initial_monmap 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 10 monclient: monmap: 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:07.561 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 1 librados: starting msgr at 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 1 librados: starting objecter 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 1 Processor -- start 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 1 -- start start 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 1 librados: setting wanted keys 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 1 librados: calling monclient init 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.559+0000 7fd7ab30e640 10 monclient: init 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a40a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a413f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:07.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a41430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a4136a50 con 0x7fd7a40a98f0 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40b79c0 con 0x7fd7a41430c0 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40b7ca0 con 0x7fd7a413f710 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.560+0000 7fd7ab30e640 10 monclient(hunting): authenticate will time out at 6310.787940s 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.562+0000 7fd7a98ab640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd7a41430c0 0x7fd7a41e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:49018/0 (socket says 172.21.15.44:49018) 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.562+0000 7fd7a98ab640 1 -- 172.21.15.44:0/3773627270 learned_addr learned my addr 172.21.15.44:0/3773627270 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.562+0000 7fd799ffb640 10 client.?.objecter ms_handle_connect 0x7fd7a41430c0 2024-03-25T17:45:07.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.562+0000 7fd799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:07.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.572+0000 7fd799ffb640 10 client.?.objecter ms_handle_connect 0x7fd7a413f710 2024-03-25T17:45:07.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.572+0000 7fd799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:07.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.574+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216989365 0 0) 0x7fd7a40b79c0 con 0x7fd7a41430c0 2024-03-25T17:45:07.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.574+0000 7fd799ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:45:07.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.574+0000 7fd799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:07.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.574+0000 7fd799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:07.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.574+0000 7fd799ffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T17:45:07.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.574+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788004eb0 con 0x7fd7a41430c0 2024-03-25T17:45:07.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.580+0000 7fd799ffb640 10 client.?.objecter ms_handle_connect 0x7fd7a40a98f0 2024-03-25T17:45:07.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.580+0000 7fd799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:07.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.584+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 973283066 0 0) 0x7fd788004eb0 con 0x7fd7a41430c0 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.584+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a40b79c0 con 0x7fd7a41430c0 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.584+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd79c001ec0 con 0x7fd7a41430c0 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.584+0000 7fd799ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.584+0000 7fd799ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.584+0000 7fd799ffb640 10 monclient(hunting): dump: 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:07.585 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:07.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.584+0000 7fd799ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:45:07.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.586+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036349708 0 0) 0x7fd7a4136a50 con 0x7fd7a40a98f0 2024-03-25T17:45:07.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.586+0000 7fd799ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:45:07.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.586+0000 7fd799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:07.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.586+0000 7fd799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:07.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.586+0000 7fd799ffb640 10 monclient(hunting): my global_id is 4506 2024-03-25T17:45:07.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.587+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788004eb0 con 0x7fd7a40a98f0 2024-03-25T17:45:07.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.591+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4129159702 0 0) 0x7fd7a40b79c0 con 0x7fd7a41430c0 2024-03-25T17:45:07.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 >> v1:172.21.15.44:6790/0 conn(0x7fd7a413f710 legacy=0x7fd7a4135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 >> v1:172.21.15.44:6789/0 conn(0x7fd7a40a98f0 legacy=0x7fd7a4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 1 monclient: found mon.b 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a4136ca0 con 0x7fd7a41430c0 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 10 monclient: _finish_auth 0 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 5 monclient: authenticate success, global_id 4489 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 10 monclient: _renew_subs 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a4136f80 con 0x7fd7a41430c0 2024-03-25T17:45:07.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:45:07.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 10 monclient: _renew_subs 2024-03-25T17:45:07.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:07.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a41360e0 con 0x7fd7a41430c0 2024-03-25T17:45:07.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.592+0000 7fd7ab30e640 1 librados: init done 2024-03-25T17:45:07.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.593+0000 7fd7ab30e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:07.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.593+0000 7fd7ab30e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:07.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.593+0000 7fd7ab30e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:07.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.593+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a40627b0 con 0x7fd7a41430c0 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.631+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd79c004e10 con 0x7fd7a41430c0 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.631+0000 7fd799ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.631+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd79c005120 con 0x7fd7a41430c0 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.631+0000 7fd799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.631+0000 7fd799ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.631+0000 7fd799ffb640 10 monclient: dump: 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:07.632 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:07.633 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:07.633 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:07.633 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:07.633 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:07.633 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:07.633 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:07.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.631+0000 7fd79b7fe640 4 set_mon_vals no callback set 2024-03-25T17:45:07.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.632+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fd79c01b0f0 con 0x7fd7a41430c0 2024-03-25T17:45:07.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.633+0000 7fd7a90aa640 10 monclient: get_auth_request con 0x7fd78806dd90 auth_method 0 2024-03-25T17:45:07.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.633+0000 7fd799ffb640 10 client.4489.objecter ms_handle_connect 0x7fd78806dd90 2024-03-25T17:45:07.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.635+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fd79c054b60 con 0x7fd7a41430c0 2024-03-25T17:45:07.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.635+0000 7fd799ffb640 10 client.4489.objecter ms_dispatch 0x7fd7a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:45:07.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.635+0000 7fd799ffb640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:45:07.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.635+0000 7fd799ffb640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:45:07.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.635+0000 7fd799ffb640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T17:45:07.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.885+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd79c050670 con 0x7fd7a41430c0 2024-03-25T17:45:07.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.885+0000 7fd799ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:07.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:07.885+0000 7fd799ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:45:08.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.138+0000 7fd7ab30e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:08.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.138+0000 7fd7ab30e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:08.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.138+0000 7fd7ab30e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:08.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.138+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7a4135070 con 0x7fd7a41430c0 2024-03-25T17:45:08.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.157+0000 7fd799ffb640 1 -- 172.21.15.44:0/3773627270 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fd79c0525a0 con 0x7fd7a41430c0 2024-03-25T17:45:08.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.157+0000 7fd799ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:08.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.157+0000 7fd799ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:45:08.158 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:45:08.158 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 10 librados: watch_flush enter 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 10 librados: watch_flush exit 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 10 client.4489.objecter successfully canceled tick 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 >> v1:172.21.15.44:6800/3257043132 conn(0x7fd78806dd90 legacy=0x7fd788070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 10 monclient: shutdown 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 >> v1:172.21.15.111:6789/0 conn(0x7fd7a41430c0 legacy=0x7fd7a41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:08.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 shutdown_connections 2024-03-25T17:45:08.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 >> 172.21.15.44:0/3773627270 conn(0x7fd7a40b7ea0 msgr2=0x7fd7a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:08.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 shutdown_connections 2024-03-25T17:45:08.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 1 -- 172.21.15.44:0/3773627270 wait complete. 2024-03-25T17:45:08.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:08.159+0000 7fd7ab30e640 1 librados: shutdown 2024-03-25T17:45:08.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:45:14.978 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.091+0000 7f926ed32640 10 monclient: get_monmap_and_config 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.091+0000 7f926ed32640 10 monclient: build_initial_monmap 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.091+0000 7f926ed32640 10 monclient: monmap: 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:45:15.092518+0000 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:45:15.092518+0000 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:15.092 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:15.093 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:45:15.093 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:45:15.093 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:15.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.092+0000 7f926ed32640 1 Processor -- start 2024-03-25T17:45:15.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.092+0000 7f926ed32640 1 -- start start 2024-03-25T17:45:15.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.092+0000 7f926ed32640 10 monclient: init 2024-03-25T17:45:15.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:15.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:45:15.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 10 monclient(hunting): picked mon.noname-c con 0x7f92680b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:15.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 10 monclient(hunting): picked mon.noname-b con 0x7f92680b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:15.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 10 monclient(hunting): picked mon.noname-a con 0x7f926814be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:15.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680af940 con 0x7f926814be90 2024-03-25T17:45:15.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680b0a70 con 0x7f92680b4d60 2024-03-25T17:45:15.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680b0d50 con 0x7f92680b1730 2024-03-25T17:45:15.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:15.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.093+0000 7f926ed32640 10 monclient(hunting): authenticate will time out at 6318.321135s 2024-03-25T17:45:15.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.099+0000 7f926cace640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f92680b4d60 0x7f926813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:53784/0 (socket says 172.21.15.44:53784) 2024-03-25T17:45:15.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.099+0000 7f926cace640 1 -- 172.21.15.44:0/3717538462 learned_addr learned my addr 172.21.15.44:0/3717538462 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:15.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.113+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102025978 0 0) 0x7f92680b0a70 con 0x7f92680b4d60 2024-03-25T17:45:15.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.113+0000 7f92677fe640 10 monclient(hunting): _negotiate 2024-03-25T17:45:15.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.113+0000 7f92677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:15.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.113+0000 7f92677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:15.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.113+0000 7f92677fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T17:45:15.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.113+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9258003520 con 0x7f92680b4d60 2024-03-25T17:45:15.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.132+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212441258 0 0) 0x7f92680b0d50 con 0x7f92680b1730 2024-03-25T17:45:15.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.132+0000 7f92677fe640 10 monclient(hunting): _negotiate 2024-03-25T17:45:15.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.132+0000 7f92677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:15.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.132+0000 7f92677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:15.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.132+0000 7f92677fe640 10 monclient(hunting): my global_id is 4496 2024-03-25T17:45:15.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.132+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92680b0a70 con 0x7f92680b1730 2024-03-25T17:45:15.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270928254 0 0) 0x7f9258003520 con 0x7f92680b4d60 2024-03-25T17:45:15.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 >> v1:172.21.15.44:6790/0 conn(0x7f92680b1730 legacy=0x7f92680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:15.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 >> v1:172.21.15.44:6789/0 conn(0x7f926814be90 legacy=0x7f926814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:15.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 10 monclient: _finish_hunting 0 2024-03-25T17:45:15.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 1 monclient: found mon.noname-c 2024-03-25T17:45:15.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680b1030 con 0x7f92680b4d60 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 10 monclient: _finish_auth 0 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 10 monclient: _check_auth_tickets 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f925c001690 con 0x7f92680b4d60 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f926ed32640 5 monclient: authenticate success, global_id 4492 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f926ed32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f92677fe640 10 monclient: dump: 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:15.135 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.136 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.136 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:15.136 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:15.136 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:15.136 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:15.136 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:15.136 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:15.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.133+0000 7f926ed32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:15.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.152+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f925c0017f0 con 0x7f92680b4d60 2024-03-25T17:45:15.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.152+0000 7f92677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:15.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.152+0000 7f92677fe640 1 -- 172.21.15.44:0/3717538462 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f925c001c00 con 0x7f92680b4d60 2024-03-25T17:45:15.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.152+0000 7f92677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:15.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.152+0000 7f92677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.152+0000 7f92677fe640 10 monclient: dump: 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.152+0000 7f926ed32640 10 monclient: get_monmap_and_config success 2024-03-25T17:45:15.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.152+0000 7f926ed32640 4 set_mon_vals no callback set 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.153+0000 7f926ed32640 10 monclient: shutdown 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.153+0000 7f926ed32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.153+0000 7f926ed32640 1 -- 172.21.15.44:0/3717538462 >> v1:172.21.15.111:6789/0 conn(0x7f92680b4d60 legacy=0x7f926813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.153+0000 7f926ed32640 1 -- 172.21.15.44:0/3717538462 shutdown_connections 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.153+0000 7f926ed32640 1 -- 172.21.15.44:0/3717538462 >> 172.21.15.44:0/3717538462 conn(0x7f92680617a0 msgr2=0x7f9268061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.153+0000 7f926ed32640 1 -- 172.21.15.44:0/3717538462 shutdown_connections 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.153+0000 7f926ed32640 1 -- 172.21.15.44:0/3717538462 wait complete. 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 10 monclient: build_initial_monmap 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 10 monclient: monmap: 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:45:15.155 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 1 librados: starting msgr at 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 1 librados: starting objecter 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 1 Processor -- start 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 1 -- start start 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 1 librados: setting wanted keys 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 1 librados: calling monclient init 2024-03-25T17:45:15.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.154+0000 7f926ed32640 10 monclient: init 2024-03-25T17:45:15.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:15.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:45:15.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 10 monclient(hunting): picked mon.noname-b con 0x7f92680b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:15.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 10 monclient(hunting): picked mon.noname-c con 0x7f92680b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:15.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 10 monclient(hunting): picked mon.noname-a con 0x7f926814be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:15.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9268140380 con 0x7f926814be90 2024-03-25T17:45:15.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9268141b50 con 0x7f92680b1730 2024-03-25T17:45:15.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9268140810 con 0x7f92680b4d60 2024-03-25T17:45:15.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:15.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.155+0000 7f926ed32640 10 monclient(hunting): authenticate will time out at 6318.383602s 2024-03-25T17:45:15.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.175+0000 7f926cace640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f92680b1730 0x7f92681df3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:53834/0 (socket says 172.21.15.44:53834) 2024-03-25T17:45:15.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.175+0000 7f926cace640 1 -- 172.21.15.44:0/3527984112 learned_addr learned my addr 172.21.15.44:0/3527984112 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:15.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.176+0000 7f92657fa640 10 client.?.objecter ms_handle_connect 0x7f92680b1730 2024-03-25T17:45:15.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.176+0000 7f92657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:15.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.182+0000 7f92657fa640 10 client.?.objecter ms_handle_connect 0x7f92680b4d60 2024-03-25T17:45:15.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.182+0000 7f92657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:15.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.183+0000 7f92657fa640 10 client.?.objecter ms_handle_connect 0x7f926814be90 2024-03-25T17:45:15.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.183+0000 7f92657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651358576 0 0) 0x7f9268141b50 con 0x7f92680b1730 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 10 monclient(hunting): my global_id is 4495 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f924c004eb0 con 0x7f92680b1730 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673295496 0 0) 0x7f9268140810 con 0x7f92680b4d60 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:15.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:15.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 10 monclient(hunting): my global_id is 4499 2024-03-25T17:45:15.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.197+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9268141b50 con 0x7f92680b4d60 2024-03-25T17:45:15.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.207+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246698851 0 0) 0x7f9268140380 con 0x7f926814be90 2024-03-25T17:45:15.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.207+0000 7f92657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:45:15.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.207+0000 7f92657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:15.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.207+0000 7f92657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:15.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.207+0000 7f92657fa640 10 monclient(hunting): my global_id is 4512 2024-03-25T17:45:15.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.207+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9268140810 con 0x7f926814be90 2024-03-25T17:45:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.210+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2765404143 0 0) 0x7f924c004eb0 con 0x7f92680b1730 2024-03-25T17:45:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.210+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9268140380 con 0x7f92680b1730 2024-03-25T17:45:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.210+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f925c001bc0 con 0x7f92680b1730 2024-03-25T17:45:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.210+0000 7f92657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.210+0000 7f92657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:45:15.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.210+0000 7f92657fa640 10 monclient(hunting): dump: 2024-03-25T17:45:15.211 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:15.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.210+0000 7f92657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.216+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3120498574 0 0) 0x7f9268141b50 con 0x7f92680b4d60 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.216+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f924c004eb0 con 0x7f92680b4d60 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.216+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9254001930 con 0x7f92680b4d60 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.216+0000 7f92657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.216+0000 7f92657fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.216+0000 7f92657fa640 10 monclient(hunting): dump: 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.217 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.218 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:15.218 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:15.218 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:15.218 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:15.218 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:15.218 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:15.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.222+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3834854206 0 0) 0x7f9268140810 con 0x7f926814be90 2024-03-25T17:45:15.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.222+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9268141b50 con 0x7f926814be90 2024-03-25T17:45:15.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.222+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9260001a90 con 0x7f926814be90 2024-03-25T17:45:15.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.222+0000 7f92657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:15.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.222+0000 7f92657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:45:15.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.222+0000 7f92657fa640 10 monclient(hunting): dump: 2024-03-25T17:45:15.223 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:15.223 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:15.224 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.224 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.224 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:15.224 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:15.224 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:15.224 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:15.224 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:15.224 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:15.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 917527223 0 0) 0x7f9268140380 con 0x7f92680b1730 2024-03-25T17:45:15.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 >> v1:172.21.15.44:6790/0 conn(0x7f92680b4d60 legacy=0x7f92681dfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:15.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 >> v1:172.21.15.44:6789/0 conn(0x7f926814be90 legacy=0x7f92681e6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:15.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 10 monclient: _finish_hunting 0 2024-03-25T17:45:15.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 1 monclient: found mon.b 2024-03-25T17:45:15.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:15.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9268141dc0 con 0x7f92680b1730 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 10 monclient: _finish_auth 0 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 10 monclient: _check_auth_tickets 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.228+0000 7f92657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 5 monclient: authenticate success, global_id 4495 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 10 monclient: _renew_subs 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92681420a0 con 0x7f92680b1730 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 10 monclient: _renew_subs 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9268138420 con 0x7f92680b1730 2024-03-25T17:45:15.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.229+0000 7f926ed32640 1 librados: init done 2024-03-25T17:45:15.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.230+0000 7f926ed32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:15.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.230+0000 7f926ed32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:15.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.230+0000 7f926ed32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:15.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.230+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92680627b0 con 0x7f92680b1730 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.512+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f925c004a90 con 0x7f92680b1730 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.512+0000 7f92657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.512+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f925c004da0 con 0x7f92680b1730 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.512+0000 7f92657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.512+0000 7f92657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.512+0000 7f92657fa640 10 monclient: dump: 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:15.513 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:15.514 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:15.514 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:15.514 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:15.514 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:15.514 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:15.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.512+0000 7f9266ffd640 4 set_mon_vals no callback set 2024-03-25T17:45:15.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.513+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f925c01ad70 con 0x7f92680b1730 2024-03-25T17:45:15.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.513+0000 7f9267fff640 10 monclient: get_auth_request con 0x7f924c06e0f0 auth_method 0 2024-03-25T17:45:15.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.513+0000 7f92657fa640 10 client.4495.objecter ms_handle_connect 0x7f924c06e0f0 2024-03-25T17:45:15.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.518+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f925c0548e0 con 0x7f92680b1730 2024-03-25T17:45:15.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.518+0000 7f92657fa640 10 client.4495.objecter ms_dispatch 0x7f9268000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:45:15.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.518+0000 7f92657fa640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:45:15.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.518+0000 7f92657fa640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:45:15.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.518+0000 7f92657fa640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T17:45:15.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.518+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f925c0503f0 con 0x7f92680b1730 2024-03-25T17:45:15.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.518+0000 7f92657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:15.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.518+0000 7f92657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:45:15.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.776+0000 7f926ed32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:15.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.776+0000 7f926ed32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:15.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.776+0000 7f926ed32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:15.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.776+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9268145010 con 0x7f92680b1730 2024-03-25T17:45:15.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.840+0000 7f92657fa640 1 -- 172.21.15.44:0/3527984112 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f925c052320 con 0x7f92680b1730 2024-03-25T17:45:15.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.840+0000 7f92657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:15.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.840+0000 7f92657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:45:15.841 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:45:15.841 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:45:15.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.842+0000 7f926ed32640 10 librados: watch_flush enter 2024-03-25T17:45:15.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.842+0000 7f926ed32640 10 librados: watch_flush exit 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.842+0000 7f926ed32640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.842+0000 7f926ed32640 10 client.4495.objecter successfully canceled tick 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 >> v1:172.21.15.44:6800/3257043132 conn(0x7f924c06e0f0 legacy=0x7f924c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 10 monclient: shutdown 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 >> v1:172.21.15.111:6789/0 conn(0x7f92680b1730 legacy=0x7f92681df3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 shutdown_connections 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 >> 172.21.15.44:0/3527984112 conn(0x7f92680617a0 msgr2=0x7f926813c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 shutdown_connections 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 1 -- 172.21.15.44:0/3527984112 wait complete. 2024-03-25T17:45:15.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:15.843+0000 7f926ed32640 1 librados: shutdown 2024-03-25T17:45:15.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:45:18.213 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:35.568 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:18.223 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:35.578 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:18.256 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:35.612 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:18.270 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:35.626 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:18.284 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:35.639 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:18.292 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:35.612 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:18.303 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:35.623 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:18.314 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:35.669 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:18.332 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:35.652 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:22.663 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:45:22.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.777+0000 7f09e8e50640 10 monclient: get_monmap_and_config 2024-03-25T17:45:22.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.777+0000 7f09e8e50640 10 monclient: build_initial_monmap 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.777+0000 7f09e8e50640 10 monclient: monmap: 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:45:22.778093+0000 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:45:22.778093+0000 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:45:22.778 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:22.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.778+0000 7f09e8e50640 1 Processor -- start 2024-03-25T17:45:22.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.778+0000 7f09e8e50640 1 -- start start 2024-03-25T17:45:22.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.778+0000 7f09e8e50640 10 monclient: init 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 10 monclient(hunting): picked mon.noname-b con 0x7f09e406f380 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 10 monclient(hunting): picked mon.noname-c con 0x7f09e406f760 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 10 monclient(hunting): picked mon.noname-a con 0x7f09e4072670 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e40718b0 con 0x7f09e4072670 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e413efc0 con 0x7f09e406f760 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e4070120 con 0x7f09e406f380 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:22.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.779+0000 7f09e8e50640 10 monclient(hunting): authenticate will time out at 6326.006946s 2024-03-25T17:45:22.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.791+0000 7f09e1d9b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f09e406f760 0x7f09e4071f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:36100/0 (socket says 172.21.15.44:36100) 2024-03-25T17:45:22.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.791+0000 7f09e1d9b640 1 -- 172.21.15.44:0/1591950250 learned_addr learned my addr 172.21.15.44:0/1591950250 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:22.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.803+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411812598 0 0) 0x7f09e413efc0 con 0x7f09e406f760 2024-03-25T17:45:22.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.803+0000 7f09e159a640 10 monclient(hunting): _negotiate 2024-03-25T17:45:22.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.803+0000 7f09e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:22.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.803+0000 7f09e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:22.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.803+0000 7f09e159a640 10 monclient(hunting): my global_id is 4498 2024-03-25T17:45:22.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.803+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d0003520 con 0x7f09e406f760 2024-03-25T17:45:22.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.804+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995750764 0 0) 0x7f09e4070120 con 0x7f09e406f380 2024-03-25T17:45:22.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.804+0000 7f09e159a640 10 monclient(hunting): _negotiate 2024-03-25T17:45:22.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.804+0000 7f09e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:22.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.804+0000 7f09e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:22.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.804+0000 7f09e159a640 10 monclient(hunting): my global_id is 4502 2024-03-25T17:45:22.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.804+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e413efc0 con 0x7f09e406f380 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.812+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 290153773 0 0) 0x7f09d0003520 con 0x7f09e406f760 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 >> v1:172.21.15.44:6790/0 conn(0x7f09e406f380 legacy=0x7f09e413f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 >> v1:172.21.15.44:6789/0 conn(0x7f09e4072670 legacy=0x7f09e414a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 10 monclient: _finish_hunting 0 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 1 monclient: found mon.noname-c 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4070610 con 0x7f09e406f760 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 10 monclient: _finish_auth 0 2024-03-25T17:45:22.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 10 monclient: _check_auth_tickets 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09cc001690 con 0x7f09e406f760 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e159a640 10 monclient: dump: 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:22.815 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:22.816 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:22.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e8e50640 5 monclient: authenticate success, global_id 4498 2024-03-25T17:45:22.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.813+0000 7f09e8e50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.827+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09cc0017f0 con 0x7f09e406f760 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.827+0000 7f09e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.827+0000 7f09e159a640 1 -- 172.21.15.44:0/1591950250 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09cc001b80 con 0x7f09e406f760 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.827+0000 7f09e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.827+0000 7f09e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.827+0000 7f09e159a640 10 monclient: dump: 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:22.828 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:22.829 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:22.829 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:22.829 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:22.829 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:22.829 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:22.829 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:22.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.827+0000 7f09e8e50640 10 monclient: get_monmap_and_config success 2024-03-25T17:45:22.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.827+0000 7f09e8e50640 4 set_mon_vals no callback set 2024-03-25T17:45:22.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.828+0000 7f09e8e50640 10 monclient: shutdown 2024-03-25T17:45:22.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.828+0000 7f09e8e50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:22.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.828+0000 7f09e8e50640 1 -- 172.21.15.44:0/1591950250 >> v1:172.21.15.111:6789/0 conn(0x7f09e406f760 legacy=0x7f09e4071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:22.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.828+0000 7f09e8e50640 1 -- 172.21.15.44:0/1591950250 shutdown_connections 2024-03-25T17:45:22.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.828+0000 7f09e8e50640 1 -- 172.21.15.44:0/1591950250 >> 172.21.15.44:0/1591950250 conn(0x7f09e406c830 msgr2=0x7f09e40b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:22.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.828+0000 7f09e8e50640 1 -- 172.21.15.44:0/1591950250 shutdown_connections 2024-03-25T17:45:22.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.828+0000 7f09e8e50640 1 -- 172.21.15.44:0/1591950250 wait complete. 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 10 monclient: build_initial_monmap 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 10 monclient: monmap: 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 1 librados: starting msgr at 2024-03-25T17:45:22.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 1 librados: starting objecter 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 1 Processor -- start 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 1 -- start start 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 1 librados: setting wanted keys 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 1 librados: calling monclient init 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.829+0000 7f09e8e50640 10 monclient: init 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 10 monclient(hunting): picked mon.noname-a con 0x7f09e406f380 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 10 monclient(hunting): picked mon.noname-b con 0x7f09e406f760 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 10 monclient(hunting): picked mon.noname-c con 0x7f09e4072670 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e41399c0 con 0x7f09e406f380 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e41346f0 con 0x7f09e406f760 2024-03-25T17:45:22.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e4139ca0 con 0x7f09e4072670 2024-03-25T17:45:22.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:22.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.830+0000 7f09e8e50640 10 monclient(hunting): authenticate will time out at 6326.058202s 2024-03-25T17:45:22.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.838+0000 7f09e1d9b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f09e406f760 0x7f09e41338d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:36146/0 (socket says 172.21.15.44:36146) 2024-03-25T17:45:22.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.838+0000 7f09e1d9b640 1 -- 172.21.15.44:0/2653503528 learned_addr learned my addr 172.21.15.44:0/2653503528 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:22.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.841+0000 7f09daffd640 10 client.?.objecter ms_handle_connect 0x7f09e406f380 2024-03-25T17:45:22.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.841+0000 7f09daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:22.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.841+0000 7f09daffd640 10 client.?.objecter ms_handle_connect 0x7f09e406f760 2024-03-25T17:45:22.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.841+0000 7f09daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:22.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.842+0000 7f09daffd640 10 client.?.objecter ms_handle_connect 0x7f09e4072670 2024-03-25T17:45:22.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.842+0000 7f09daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:22.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.853+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308611837 0 0) 0x7f09e41346f0 con 0x7f09e406f760 2024-03-25T17:45:22.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.853+0000 7f09daffd640 10 monclient(hunting): _negotiate 2024-03-25T17:45:22.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.853+0000 7f09daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:22.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.853+0000 7f09daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:22.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.853+0000 7f09daffd640 10 monclient(hunting): my global_id is 4501 2024-03-25T17:45:22.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.853+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c4004eb0 con 0x7f09e406f760 2024-03-25T17:45:22.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.858+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371355658 0 0) 0x7f09e41399c0 con 0x7f09e406f380 2024-03-25T17:45:22.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.858+0000 7f09daffd640 10 monclient(hunting): _negotiate 2024-03-25T17:45:22.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.858+0000 7f09daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:22.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.858+0000 7f09daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:22.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.858+0000 7f09daffd640 10 monclient(hunting): my global_id is 4518 2024-03-25T17:45:22.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.858+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e41346f0 con 0x7f09e406f380 2024-03-25T17:45:22.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.868+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 367566217 0 0) 0x7f09c4004eb0 con 0x7f09e406f760 2024-03-25T17:45:22.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.868+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09e41399c0 con 0x7f09e406f760 2024-03-25T17:45:22.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.868+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09cc001bc0 con 0x7f09e406f760 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.868+0000 7f09daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.868+0000 7f09daffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.868+0000 7f09daffd640 10 monclient(hunting): dump: 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:22.869 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:22.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.868+0000 7f09daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:45:22.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.871+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855090625 0 0) 0x7f09e4139ca0 con 0x7f09e4072670 2024-03-25T17:45:22.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.871+0000 7f09daffd640 10 monclient(hunting): _negotiate 2024-03-25T17:45:22.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.871+0000 7f09daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:22.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.871+0000 7f09daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:22.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.871+0000 7f09daffd640 10 monclient(hunting): my global_id is 4505 2024-03-25T17:45:22.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.871+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c4004eb0 con 0x7f09e4072670 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.873+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797375228 0 0) 0x7f09e41346f0 con 0x7f09e406f380 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.873+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09e4139ca0 con 0x7f09e406f380 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.873+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09d4001930 con 0x7f09e406f380 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.873+0000 7f09daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.873+0000 7f09daffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.873+0000 7f09daffd640 10 monclient(hunting): dump: 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:22.874 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:22.875 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:22.875 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:22.875 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:22.875 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:22.875 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:22.875 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:22.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1963445554 0 0) 0x7f09e4139ca0 con 0x7f09e406f380 2024-03-25T17:45:22.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 >> v1:172.21.15.44:6790/0 conn(0x7f09e4072670 legacy=0x7f09e4133fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:22.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 >> v1:172.21.15.111:6789/0 conn(0x7f09e406f760 legacy=0x7f09e41338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:22.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 10 monclient: _finish_hunting 0 2024-03-25T17:45:22.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 1 monclient: found mon.a 2024-03-25T17:45:22.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4137f00 con 0x7f09e406f380 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 10 monclient: _finish_auth 0 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 10 monclient: _check_auth_tickets 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 5 monclient: authenticate success, global_id 4518 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 10 monclient: _renew_subs 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e406b740 con 0x7f09e406f380 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 10 monclient: _renew_subs 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e41db330 con 0x7f09e406f380 2024-03-25T17:45:22.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.880+0000 7f09e8e50640 1 librados: init done 2024-03-25T17:45:22.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.881+0000 7f09e8e50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:22.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.881+0000 7f09e8e50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:22.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.881+0000 7f09e8e50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:22.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:22.881+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e4070b00 con 0x7f09e406f380 2024-03-25T17:45:23.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.210+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09d4003610 con 0x7f09e406f380 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.210+0000 7f09daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.210+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09d40039a0 con 0x7f09e406f380 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.210+0000 7f09daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.210+0000 7f09daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.210+0000 7f09daffd640 10 monclient: dump: 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:23.211 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:23.212 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:23.212 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:23.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.210+0000 7f09e0d99640 4 set_mon_vals no callback set 2024-03-25T17:45:23.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.210+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f09d4003bc0 con 0x7f09e406f380 2024-03-25T17:45:23.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.211+0000 7f09e1d9b640 10 monclient: get_auth_request con 0x7f09c406df70 auth_method 0 2024-03-25T17:45:23.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.211+0000 7f09daffd640 10 client.4518.objecter ms_handle_connect 0x7f09c406df70 2024-03-25T17:45:23.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.220+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f09d4053530 con 0x7f09e406f380 2024-03-25T17:45:23.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.221+0000 7f09daffd640 10 client.4518.objecter ms_dispatch 0x7f09e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:45:23.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.221+0000 7f09daffd640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:45:23.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.221+0000 7f09daffd640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:45:23.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.221+0000 7f09daffd640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-25T17:45:23.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.221+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09d404f040 con 0x7f09e406f380 2024-03-25T17:45:23.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.221+0000 7f09daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:23.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.221+0000 7f09daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:45:23.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.476+0000 7f09e8e50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:23.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.476+0000 7f09e8e50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:23.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.476+0000 7f09e8e50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:23.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.476+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09e40627b0 con 0x7f09e406f380 2024-03-25T17:45:23.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.507+0000 7f09daffd640 1 -- 172.21.15.44:0/2653503528 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f09d4050f70 con 0x7f09e406f380 2024-03-25T17:45:23.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.507+0000 7f09daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:23.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.507+0000 7f09daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:45:23.508 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:45:23.508 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 10 librados: watch_flush enter 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 10 librados: watch_flush exit 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 10 client.4518.objecter successfully canceled tick 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 >> v1:172.21.15.44:6800/3257043132 conn(0x7f09c406df70 legacy=0x7f09c4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 10 monclient: shutdown 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 >> v1:172.21.15.44:6789/0 conn(0x7f09e406f380 legacy=0x7f09e41daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 shutdown_connections 2024-03-25T17:45:23.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 >> 172.21.15.44:0/2653503528 conn(0x7f09e406c830 msgr2=0x7f09e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:23.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 shutdown_connections 2024-03-25T17:45:23.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 1 -- 172.21.15.44:0/2653503528 wait complete. 2024-03-25T17:45:23.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:23.510+0000 7f09e8e50640 1 librados: shutdown 2024-03-25T17:45:23.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:45:30.332 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.432+0000 7f8cb1893640 10 monclient: get_monmap_and_config 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.432+0000 7f8cb1893640 10 monclient: build_initial_monmap 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.432+0000 7f8cb1893640 10 monclient: monmap: 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:45:30.433818+0000 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:45:30.433818+0000 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:30.433 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:30.434 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:45:30.434 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:45:30.434 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:30.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.433+0000 7f8cb1893640 1 Processor -- start 2024-03-25T17:45:30.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.433+0000 7f8cb1893640 1 -- start start 2024-03-25T17:45:30.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.433+0000 7f8cb1893640 10 monclient: init 2024-03-25T17:45:30.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:30.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:45:30.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 10 monclient(hunting): picked mon.noname-c con 0x7f8cac0b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:30.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 10 monclient(hunting): picked mon.noname-a con 0x7f8cac0b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:30.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 10 monclient(hunting): picked mon.noname-b con 0x7f8cac14be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:30.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0af940 con 0x7f8cac0b1730 2024-03-25T17:45:30.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0b0a70 con 0x7f8cac0b4d60 2024-03-25T17:45:30.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0b0d50 con 0x7f8cac14be90 2024-03-25T17:45:30.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:30.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8cb1893640 10 monclient(hunting): authenticate will time out at 6333.661819s 2024-03-25T17:45:30.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8caaffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8cac0b4d60 0x7f8cac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:42428/0 (socket says 172.21.15.44:42428) 2024-03-25T17:45:30.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.434+0000 7f8caaffd640 1 -- 172.21.15.44:0/1012395932 learned_addr learned my addr 172.21.15.44:0/1012395932 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:30.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.459+0000 7f8caa7fc640 1 -- 172.21.15.44:0/1012395932 >> v1:172.21.15.44:6789/0 conn(0x7f8cac0b1730 legacy=0x7f8cac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T17:45:30.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.459+0000 7f8caa7fc640 1 -- 172.21.15.44:0/1012395932 >> v1:172.21.15.44:6789/0 conn(0x7f8cac0b1730 legacy=0x7f8cac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T17:45:30.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.459+0000 7f8caa7fc640 1 --1- 172.21.15.44:0/1012395932 >> v1:172.21.15.44:6789/0 conn(0x7f8cac0b1730 0x7f8cac0b1be0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T17:45:30.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.460+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995523529 0 0) 0x7f8cac0b0d50 con 0x7f8cac14be90 2024-03-25T17:45:30.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.461+0000 7f8ca9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:45:30.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.461+0000 7f8ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:30.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.461+0000 7f8ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:30.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.461+0000 7f8ca9ffb640 10 monclient(hunting): my global_id is 4508 2024-03-25T17:45:30.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.461+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c94003520 con 0x7f8cac14be90 2024-03-25T17:45:30.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139093042 0 0) 0x7f8c94003520 con 0x7f8cac14be90 2024-03-25T17:45:30.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 >> v1:172.21.15.111:6789/0 conn(0x7f8cac0b4d60 legacy=0x7f8cac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:30.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 >> v1:172.21.15.44:6789/0 conn(0x7f8cac0b1730 legacy=0x7f8cac0b1be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:45:30.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:45:30.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 1 monclient: found mon.noname-b 2024-03-25T17:45:30.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:30.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:45:30.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac0b1030 con 0x7f8cac14be90 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 10 monclient: _finish_auth 0 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8ca4001a10 con 0x7f8cac14be90 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8cb1893640 5 monclient: authenticate success, global_id 4508 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8cb1893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8ca9ffb640 10 monclient: dump: 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.471 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:30.472 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:30.472 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:30.472 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:30.472 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:30.472 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:30.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.469+0000 7f8cb1893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.490+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ca4001be0 con 0x7f8cac14be90 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.490+0000 7f8ca9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.490+0000 7f8ca9ffb640 1 -- 172.21.15.44:0/1012395932 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8ca4002030 con 0x7f8cac14be90 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.490+0000 7f8cb1893640 10 monclient: get_monmap_and_config success 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.490+0000 7f8cb1893640 4 set_mon_vals no callback set 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.490+0000 7f8ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.490+0000 7f8ca9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.490+0000 7f8ca9ffb640 10 monclient: dump: 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.492 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:30.492 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:30.492 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:30.492 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:30.492 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:30.492 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.491+0000 7f8cb1893640 10 monclient: shutdown 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.491+0000 7f8cb1893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.491+0000 7f8cb1893640 1 -- 172.21.15.44:0/1012395932 >> v1:172.21.15.44:6790/0 conn(0x7f8cac14be90 legacy=0x7f8cac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.491+0000 7f8cb1893640 1 -- 172.21.15.44:0/1012395932 shutdown_connections 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.491+0000 7f8cb1893640 1 -- 172.21.15.44:0/1012395932 >> 172.21.15.44:0/1012395932 conn(0x7f8cac0617a0 msgr2=0x7f8cac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.491+0000 7f8cb1893640 1 -- 172.21.15.44:0/1012395932 shutdown_connections 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.491+0000 7f8cb1893640 1 -- 172.21.15.44:0/1012395932 wait complete. 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 10 monclient: build_initial_monmap 2024-03-25T17:45:30.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 10 monclient: monmap: 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 1 librados: starting msgr at 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 1 librados: starting objecter 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 1 Processor -- start 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 1 -- start start 2024-03-25T17:45:30.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 1 librados: setting wanted keys 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 1 librados: calling monclient init 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.492+0000 7f8cb1893640 10 monclient: init 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 10 monclient(hunting): picked mon.noname-c con 0x7f8cac0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 10 monclient(hunting): picked mon.noname-a con 0x7f8cac0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 10 monclient(hunting): picked mon.noname-b con 0x7f8cac14be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac142f40 con 0x7f8cac0b4d60 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac1387b0 con 0x7f8cac14be90 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac1428c0 con 0x7f8cac0b1730 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:30.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.493+0000 7f8cb1893640 10 monclient(hunting): authenticate will time out at 6333.721273s 2024-03-25T17:45:30.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.500+0000 7f8caaffd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f8cac0b1730 0x7f8cac1380a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:56710/0 (socket says 172.21.15.44:56710) 2024-03-25T17:45:30.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.500+0000 7f8caaffd640 1 -- 172.21.15.44:0/1213181733 learned_addr learned my addr 172.21.15.44:0/1213181733 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:30.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.503+0000 7f8c93fff640 10 client.?.objecter ms_handle_connect 0x7f8cac0b1730 2024-03-25T17:45:30.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.503+0000 7f8c93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:30.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.507+0000 7f8c93fff640 10 client.?.objecter ms_handle_connect 0x7f8cac0b4d60 2024-03-25T17:45:30.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.507+0000 7f8c93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:30.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.508+0000 7f8c93fff640 10 client.?.objecter ms_handle_connect 0x7f8cac14be90 2024-03-25T17:45:30.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.508+0000 7f8c93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:30.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.509+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592910433 0 0) 0x7f8cac1428c0 con 0x7f8cac0b1730 2024-03-25T17:45:30.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.510+0000 7f8c93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:30.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.510+0000 7f8c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:30.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.510+0000 7f8c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:30.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.510+0000 7f8c93fff640 10 monclient(hunting): my global_id is 4511 2024-03-25T17:45:30.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.510+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c98004eb0 con 0x7f8cac0b1730 2024-03-25T17:45:30.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.518+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48738733 0 0) 0x7f8cac142f40 con 0x7f8cac0b4d60 2024-03-25T17:45:30.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.518+0000 7f8c93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:30.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.518+0000 7f8c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:30.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.518+0000 7f8c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:30.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.518+0000 7f8c93fff640 10 monclient(hunting): my global_id is 4521 2024-03-25T17:45:30.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.518+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cac1428c0 con 0x7f8cac0b4d60 2024-03-25T17:45:30.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.520+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706403102 0 0) 0x7f8cac1387b0 con 0x7f8cac14be90 2024-03-25T17:45:30.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.520+0000 7f8c93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:30.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.520+0000 7f8c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:30.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.520+0000 7f8c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:30.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.520+0000 7f8c93fff640 10 monclient(hunting): my global_id is 4504 2024-03-25T17:45:30.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.520+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cac142f40 con 0x7f8cac14be90 2024-03-25T17:45:30.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.532+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4029195969 0 0) 0x7f8c98004eb0 con 0x7f8cac0b1730 2024-03-25T17:45:30.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.532+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cac1387b0 con 0x7f8cac0b1730 2024-03-25T17:45:30.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.532+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8c9c0018b0 con 0x7f8cac0b1730 2024-03-25T17:45:30.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.532+0000 7f8c93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:30.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.532+0000 7f8c93fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:45:30.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.532+0000 7f8c93fff640 10 monclient(hunting): dump: 2024-03-25T17:45:30.532 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:30.532 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:30.533 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.532+0000 7f8c93fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:45:30.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.534+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2106990306 0 0) 0x7f8cac142f40 con 0x7f8cac14be90 2024-03-25T17:45:30.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.534+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c98004eb0 con 0x7f8cac14be90 2024-03-25T17:45:30.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.534+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8ca4001ef0 con 0x7f8cac14be90 2024-03-25T17:45:30.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.534+0000 7f8c93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.534+0000 7f8c93fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.534+0000 7f8c93fff640 10 monclient(hunting): dump: 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:30.535 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:30.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.536+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 958592800 0 0) 0x7f8cac1428c0 con 0x7f8cac0b4d60 2024-03-25T17:45:30.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.536+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cac142f40 con 0x7f8cac0b4d60 2024-03-25T17:45:30.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.536+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8ca0003fb0 con 0x7f8cac0b4d60 2024-03-25T17:45:30.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.537+0000 7f8c93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:30.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.537+0000 7f8c93fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:45:30.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.537+0000 7f8c93fff640 10 monclient(hunting): dump: 2024-03-25T17:45:30.537 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:30.538 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:30.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.550+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 563206260 0 0) 0x7f8c98004eb0 con 0x7f8cac14be90 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 >> v1:172.21.15.44:6790/0 conn(0x7f8cac0b1730 legacy=0x7f8cac1380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 >> v1:172.21.15.44:6789/0 conn(0x7f8cac0b4d60 legacy=0x7f8cac1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 10 monclient: _finish_hunting 0 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 1 monclient: found mon.b 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac142a50 con 0x7f8cac14be90 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 10 monclient: _finish_auth 0 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 10 monclient: _check_auth_tickets 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8c93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 5 monclient: authenticate success, global_id 4504 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 10 monclient: _renew_subs 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac1423e0 con 0x7f8cac14be90 2024-03-25T17:45:30.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:45:30.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 10 monclient: _renew_subs 2024-03-25T17:45:30.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:30.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac13b770 con 0x7f8cac14be90 2024-03-25T17:45:30.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.551+0000 7f8cb1893640 1 librados: init done 2024-03-25T17:45:30.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.552+0000 7f8cb1893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:30.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.552+0000 7f8cb1893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:30.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.552+0000 7f8cb1893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:30.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.552+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac142da0 con 0x7f8cac14be90 2024-03-25T17:45:30.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.829+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ca4001360 con 0x7f8cac14be90 2024-03-25T17:45:30.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.829+0000 7f8c93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:30.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.829+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8ca4002050 con 0x7f8cac14be90 2024-03-25T17:45:30.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.829+0000 7f8c93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:30.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.829+0000 7f8c93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.829+0000 7f8c93fff640 10 monclient: dump: 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:30.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.829+0000 7f8ca97fa640 4 set_mon_vals no callback set 2024-03-25T17:45:30.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.830+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f8ca4019a90 con 0x7f8cac14be90 2024-03-25T17:45:30.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.830+0000 7f8caaffd640 10 monclient: get_auth_request con 0x7f8c9806e0b0 auth_method 0 2024-03-25T17:45:30.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.830+0000 7f8c93fff640 10 client.4504.objecter ms_handle_connect 0x7f8c9806e0b0 2024-03-25T17:45:30.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.834+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f8ca40536b0 con 0x7f8cac14be90 2024-03-25T17:45:30.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.834+0000 7f8c93fff640 10 client.4504.objecter ms_dispatch 0x7f8cac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:45:30.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.834+0000 7f8c93fff640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:45:30.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.834+0000 7f8c93fff640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:45:30.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.834+0000 7f8c93fff640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T17:45:30.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.834+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ca404eff0 con 0x7f8cac14be90 2024-03-25T17:45:30.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.834+0000 7f8c93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:30.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:30.834+0000 7f8c93fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:45:31.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.094+0000 7f8cb1893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:31.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.094+0000 7f8cb1893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:31.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.094+0000 7f8cb1893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:31.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.094+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cac1430d0 con 0x7f8cac14be90 2024-03-25T17:45:31.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.122+0000 7f8c93fff640 1 -- 172.21.15.44:0/1213181733 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f8ca40510f0 con 0x7f8cac14be90 2024-03-25T17:45:31.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.122+0000 7f8c93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:31.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.122+0000 7f8c93fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:45:31.123 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:45:31.123 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.124+0000 7f8cb1893640 10 librados: watch_flush enter 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.124+0000 7f8cb1893640 10 librados: watch_flush exit 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.124+0000 7f8cb1893640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.124+0000 7f8cb1893640 10 client.4504.objecter successfully canceled tick 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.124+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 >> v1:172.21.15.44:6800/3257043132 conn(0x7f8c9806e0b0 legacy=0x7f8c98070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.125+0000 7f8cb1893640 10 monclient: shutdown 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.125+0000 7f8cb1893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.125+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 >> v1:172.21.15.111:6789/0 conn(0x7f8cac14be90 legacy=0x7f8cac1e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.125+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 shutdown_connections 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.125+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 >> 172.21.15.44:0/1213181733 conn(0x7f8cac0617a0 msgr2=0x7f8cac0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:31.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.125+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 shutdown_connections 2024-03-25T17:45:31.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.125+0000 7f8cb1893640 1 -- 172.21.15.44:0/1213181733 wait complete. 2024-03-25T17:45:31.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:31.125+0000 7f8cb1893640 1 librados: shutdown 2024-03-25T17:45:31.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:45:37.943 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.055+0000 7fa2cf5a5640 10 monclient: get_monmap_and_config 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.055+0000 7fa2cf5a5640 10 monclient: build_initial_monmap 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.056+0000 7fa2cf5a5640 10 monclient: monmap: 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:45:38.056913+0000 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:45:38.056913+0000 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:38.056 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:38.057 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:45:38.057 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:45:38.057 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:38.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.056+0000 7fa2cf5a5640 1 Processor -- start 2024-03-25T17:45:38.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.056+0000 7fa2cf5a5640 1 -- start start 2024-03-25T17:45:38.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.056+0000 7fa2cf5a5640 10 monclient: init 2024-03-25T17:45:38.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:38.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:45:38.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 10 monclient(hunting): picked mon.noname-a con 0x7fa2c80a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:38.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 10 monclient(hunting): picked mon.noname-b con 0x7fa2c813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:38.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 10 monclient(hunting): picked mon.noname-c con 0x7fa2c81430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:38.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c8142520 con 0x7fa2c80a98f0 2024-03-25T17:45:38.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c80b28a0 con 0x7fa2c81430c0 2024-03-25T17:45:38.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c80b2a30 con 0x7fa2c813f710 2024-03-25T17:45:38.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:38.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.057+0000 7fa2cf5a5640 10 monclient(hunting): authenticate will time out at 6341.285538s 2024-03-25T17:45:38.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.060+0000 7fa2ccb40640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fa2c813f710 0x7fa2c81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:33230/0 (socket says 172.21.15.44:33230) 2024-03-25T17:45:38.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.060+0000 7fa2ccb40640 1 -- 172.21.15.44:0/1675198355 learned_addr learned my addr 172.21.15.44:0/1675198355 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:38.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.073+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170556472 0 0) 0x7fa2c80b2a30 con 0x7fa2c813f710 2024-03-25T17:45:38.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.073+0000 7fa2bffff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:38.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.073+0000 7fa2bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:38.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.073+0000 7fa2bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:38.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.073+0000 7fa2bffff640 10 monclient(hunting): my global_id is 4514 2024-03-25T17:45:38.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.073+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b4003520 con 0x7fa2c813f710 2024-03-25T17:45:38.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 678346018 0 0) 0x7fa2b4003520 con 0x7fa2c813f710 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 >> v1:172.21.15.111:6789/0 conn(0x7fa2c81430c0 legacy=0x7fa2c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 >> v1:172.21.15.44:6789/0 conn(0x7fa2c80a98f0 legacy=0x7fa2c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 10 monclient: _finish_hunting 0 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 1 monclient: found mon.noname-b 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c80b44c0 con 0x7fa2c813f710 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 10 monclient: _finish_auth 0 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 10 monclient: _check_auth_tickets 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa2b8001690 con 0x7fa2c813f710 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:45:38.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2bffff640 10 monclient: dump: 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2cf5a5640 5 monclient: authenticate success, global_id 4514 2024-03-25T17:45:38.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.095+0000 7fa2cf5a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.107+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2b80017f0 con 0x7fa2c813f710 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.107+0000 7fa2bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.107+0000 7fa2bffff640 1 -- 172.21.15.44:0/1675198355 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa2b8001c00 con 0x7fa2c813f710 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.107+0000 7fa2cf5a5640 10 monclient: get_monmap_and_config success 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.107+0000 7fa2cf5a5640 4 set_mon_vals no callback set 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.107+0000 7fa2bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.107+0000 7fa2bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.107+0000 7fa2bffff640 10 monclient: dump: 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:38.108 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:38.109 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:38.109 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:38.109 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:38.109 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.108+0000 7fa2cf5a5640 10 monclient: shutdown 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.108+0000 7fa2cf5a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.108+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1675198355 >> v1:172.21.15.44:6790/0 conn(0x7fa2c813f710 legacy=0x7fa2c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.108+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1675198355 shutdown_connections 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.108+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1675198355 >> 172.21.15.44:0/1675198355 conn(0x7fa2c80b7ea0 msgr2=0x7fa2c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.108+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1675198355 shutdown_connections 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.108+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1675198355 wait complete. 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 10 monclient: build_initial_monmap 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 10 monclient: monmap: 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:38.111 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 1 librados: starting msgr at 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 1 librados: starting objecter 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 1 Processor -- start 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 1 -- start start 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 1 librados: setting wanted keys 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 1 librados: calling monclient init 2024-03-25T17:45:38.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.109+0000 7fa2cf5a5640 10 monclient: init 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 10 monclient(hunting): picked mon.noname-b con 0x7fa2c80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 10 monclient(hunting): picked mon.noname-a con 0x7fa2c813f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 10 monclient(hunting): picked mon.noname-c con 0x7fa2c81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c8139070 con 0x7fa2c813f710 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c8133710 con 0x7fa2c80a98f0 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c81338a0 con 0x7fa2c81430c0 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:38.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.110+0000 7fa2cf5a5640 10 monclient(hunting): authenticate will time out at 6341.338163s 2024-03-25T17:45:38.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.121+0000 7fa2ccb40640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa2c813f710 0x7fa2c81daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:59536/0 (socket says 172.21.15.44:59536) 2024-03-25T17:45:38.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.122+0000 7fa2ccb40640 1 -- 172.21.15.44:0/1765104979 learned_addr learned my addr 172.21.15.44:0/1765104979 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:38.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.125+0000 7fa2bdffb640 10 client.?.objecter ms_handle_connect 0x7fa2c813f710 2024-03-25T17:45:38.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.125+0000 7fa2bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:38.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.126+0000 7fa2bdffb640 10 client.?.objecter ms_handle_connect 0x7fa2c80a98f0 2024-03-25T17:45:38.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.126+0000 7fa2bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:38.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.129+0000 7fa2bdffb640 10 client.?.objecter ms_handle_connect 0x7fa2c81430c0 2024-03-25T17:45:38.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.129+0000 7fa2bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:38.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.190+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146101951 0 0) 0x7fa2c8139070 con 0x7fa2c813f710 2024-03-25T17:45:38.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.190+0000 7fa2bdffb640 10 monclient(hunting): _negotiate 2024-03-25T17:45:38.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.190+0000 7fa2bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:38.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.190+0000 7fa2bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:38.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.190+0000 7fa2bdffb640 10 monclient(hunting): my global_id is 4527 2024-03-25T17:45:38.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.190+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ac004eb0 con 0x7fa2c813f710 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.206+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2700923254 0 0) 0x7fa2ac004eb0 con 0x7fa2c813f710 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.206+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2c8139070 con 0x7fa2c813f710 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.206+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa2b8001bc0 con 0x7fa2c813f710 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.206+0000 7fa2bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.206+0000 7fa2bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.206+0000 7fa2bdffb640 10 monclient(hunting): dump: 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:38.207 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:38.208 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:38.208 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:38.208 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:38.208 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:38.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.206+0000 7fa2bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2745005267 0 0) 0x7fa2c8139070 con 0x7fa2c813f710 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 >> v1:172.21.15.44:6790/0 conn(0x7fa2c81430c0 legacy=0x7fa2c8133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 >> v1:172.21.15.111:6789/0 conn(0x7fa2c80a98f0 legacy=0x7fa2c81378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 10 monclient: _finish_hunting 0 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 1 monclient: found mon.a 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8133a30 con 0x7fa2c813f710 2024-03-25T17:45:38.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 10 monclient: _finish_auth 0 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 10 monclient: _check_auth_tickets 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 5 monclient: authenticate success, global_id 4527 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 10 monclient: _renew_subs 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c80b7100 con 0x7fa2c813f710 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 10 monclient: _renew_subs 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c8136e60 con 0x7fa2c813f710 2024-03-25T17:45:38.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.240+0000 7fa2cf5a5640 1 librados: init done 2024-03-25T17:45:38.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.241+0000 7fa2cf5a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:38.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.241+0000 7fa2cf5a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:38.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.241+0000 7fa2cf5a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:38.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.241+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c80627b0 con 0x7fa2c813f710 2024-03-25T17:45:38.632 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:55.988 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.656 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:55.993 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.664+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2b8004a90 con 0x7fa2c813f710 2024-03-25T17:45:38.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.664+0000 7fa2bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:38.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.664+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa2b8004da0 con 0x7fa2c813f710 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.664+0000 7fa2bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.664+0000 7fa2bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.664+0000 7fa2bdffb640 10 monclient: dump: 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:38.665 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.664+0000 7fa2bf7fe640 4 set_mon_vals no callback set 2024-03-25T17:45:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.665+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fa2b801ad70 con 0x7fa2c813f710 2024-03-25T17:45:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.666+0000 7fa2cd341640 10 monclient: get_auth_request con 0x7fa2ac06dd00 auth_method 0 2024-03-25T17:45:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.666+0000 7fa2bdffb640 10 client.4527.objecter ms_handle_connect 0x7fa2ac06dd00 2024-03-25T17:45:38.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.670+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fa2b8054920 con 0x7fa2c813f710 2024-03-25T17:45:38.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.670+0000 7fa2bdffb640 10 client.4527.objecter ms_dispatch 0x7fa2c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:45:38.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.670+0000 7fa2bdffb640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:45:38.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.670+0000 7fa2bdffb640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:45:38.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.670+0000 7fa2bdffb640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T17:45:38.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.670+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2b8050430 con 0x7fa2c813f710 2024-03-25T17:45:38.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.670+0000 7fa2bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:38.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.670+0000 7fa2bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:45:38.675 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:56.031 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.689 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:56.009 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.696 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:56.016 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.703 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:56.059 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.724 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:56.080 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.730 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:56.050 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.753 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:56.109 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:38.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.917+0000 7fa2cf5a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:38.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.917+0000 7fa2cf5a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:38.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.917+0000 7fa2cf5a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:45:38.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.917+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2c8069f40 con 0x7fa2c813f710 2024-03-25T17:45:38.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.984+0000 7fa2bdffb640 1 -- 172.21.15.44:0/1765104979 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fa2b8052360 con 0x7fa2c813f710 2024-03-25T17:45:38.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.984+0000 7fa2bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:38.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.984+0000 7fa2bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:45:38.985 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:45:38.985 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:45:38.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 10 librados: watch_flush enter 2024-03-25T17:45:38.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 10 librados: watch_flush exit 2024-03-25T17:45:38.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T17:45:38.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 10 client.4527.objecter successfully canceled tick 2024-03-25T17:45:38.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 >> v1:172.21.15.44:6800/3257043132 conn(0x7fa2ac06dd00 legacy=0x7fa2ac0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:38.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 10 monclient: shutdown 2024-03-25T17:45:38.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:38.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 >> v1:172.21.15.44:6789/0 conn(0x7fa2c813f710 legacy=0x7fa2c81daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:38.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 shutdown_connections 2024-03-25T17:45:38.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 >> 172.21.15.44:0/1765104979 conn(0x7fa2c80b7ea0 msgr2=0x7fa2c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:38.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 shutdown_connections 2024-03-25T17:45:38.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 1 -- 172.21.15.44:0/1765104979 wait complete. 2024-03-25T17:45:38.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:38.987+0000 7fa2cf5a5640 1 librados: shutdown 2024-03-25T17:45:38.999 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:45:45.807 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.929+0000 7f102a2a5640 10 monclient: get_monmap_and_config 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.929+0000 7f102a2a5640 10 monclient: build_initial_monmap 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.929+0000 7f102a2a5640 10 monclient: monmap: 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:45:45.930562+0000 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:45:45.930562+0000 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:45.930 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:45.931 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:45:45.931 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:45:45.931 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.930+0000 7f102a2a5640 1 Processor -- start 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.930+0000 7f102a2a5640 1 -- start start 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.930+0000 7f102a2a5640 10 monclient: init 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 10 monclient(hunting): picked mon.noname-a con 0x7f10240b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 10 monclient(hunting): picked mon.noname-c con 0x7f10240b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 10 monclient(hunting): picked mon.noname-b con 0x7f102414be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240af940 con 0x7f10240b4d60 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240b0a70 con 0x7f10240b1730 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240b0d50 con 0x7f102414be90 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:45.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.931+0000 7f102a2a5640 10 monclient(hunting): authenticate will time out at 6349.158855s 2024-03-25T17:45:45.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.945+0000 7f10237fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f10240b1730 0x7f10240b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:59164/0 (socket says 172.21.15.44:59164) 2024-03-25T17:45:45.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.945+0000 7f10237fe640 1 -- 172.21.15.44:0/990244257 learned_addr learned my addr 172.21.15.44:0/990244257 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:45.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.974+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084806559 0 0) 0x7f10240b0a70 con 0x7f10240b1730 2024-03-25T17:45:45.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.974+0000 7f1022ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:45:45.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.974+0000 7f1022ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:45.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.974+0000 7f1022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:45.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.974+0000 7f1022ffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T17:45:45.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.974+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f100c003520 con 0x7f10240b1730 2024-03-25T17:45:45.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.976+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744105086 0 0) 0x7f10240b0d50 con 0x7f102414be90 2024-03-25T17:45:45.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.976+0000 7f1022ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:45:45.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.976+0000 7f1022ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:45.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.976+0000 7f1022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:45.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.976+0000 7f1022ffd640 10 monclient(hunting): my global_id is 4520 2024-03-25T17:45:45.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.976+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10240b0a70 con 0x7f102414be90 2024-03-25T17:45:45.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.979+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616832413 0 0) 0x7f10240af940 con 0x7f10240b4d60 2024-03-25T17:45:45.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.979+0000 7f1022ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:45:45.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.979+0000 7f1022ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:45.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.979+0000 7f1022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:45.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.979+0000 7f1022ffd640 10 monclient(hunting): my global_id is 4530 2024-03-25T17:45:45.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.979+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10240b0d50 con 0x7f10240b4d60 2024-03-25T17:45:45.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981009083 0 0) 0x7f10240b0a70 con 0x7f102414be90 2024-03-25T17:45:45.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 >> v1:172.21.15.111:6789/0 conn(0x7f10240b1730 legacy=0x7f10240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:45.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 >> v1:172.21.15.44:6789/0 conn(0x7f10240b4d60 legacy=0x7f102413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:45.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:45:45.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 1 monclient: found mon.noname-b 2024-03-25T17:45:45.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240b1030 con 0x7f102414be90 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 10 monclient: _finish_auth 0 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f101c0019f0 con 0x7f102414be90 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f1022ffd640 10 monclient: dump: 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:45.997 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:45.998 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:45.998 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:45.998 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:45.998 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:45.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f102a2a5640 5 monclient: authenticate success, global_id 4520 2024-03-25T17:45:45.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:45.995+0000 7f102a2a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:46.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.013+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f101c001b50 con 0x7f102414be90 2024-03-25T17:45:46.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.013+0000 7f1022ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:46.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.013+0000 7f1022ffd640 1 -- 172.21.15.44:0/990244257 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f101c001f60 con 0x7f102414be90 2024-03-25T17:45:46.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.013+0000 7f102a2a5640 10 monclient: get_monmap_and_config success 2024-03-25T17:45:46.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.013+0000 7f102a2a5640 4 set_mon_vals no callback set 2024-03-25T17:45:46.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.013+0000 7f1022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:46.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.013+0000 7f1022ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:45:46.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.013+0000 7f1022ffd640 10 monclient: dump: 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:46.015 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.014+0000 7f102a2a5640 10 monclient: shutdown 2024-03-25T17:45:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.014+0000 7f102a2a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:46.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.014+0000 7f102a2a5640 1 -- 172.21.15.44:0/990244257 >> v1:172.21.15.44:6790/0 conn(0x7f102414be90 legacy=0x7f102414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.014+0000 7f102a2a5640 1 -- 172.21.15.44:0/990244257 shutdown_connections 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.014+0000 7f102a2a5640 1 -- 172.21.15.44:0/990244257 >> 172.21.15.44:0/990244257 conn(0x7f10240617a0 msgr2=0x7f1024061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.014+0000 7f102a2a5640 1 -- 172.21.15.44:0/990244257 shutdown_connections 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.014+0000 7f102a2a5640 1 -- 172.21.15.44:0/990244257 wait complete. 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.015+0000 7f102a2a5640 10 monclient: build_initial_monmap 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.015+0000 7f102a2a5640 10 monclient: monmap: 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:45:46.017 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:45:46.018 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:46.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.016+0000 7f102a2a5640 1 librados: starting msgr at 2024-03-25T17:45:46.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.016+0000 7f102a2a5640 1 librados: starting objecter 2024-03-25T17:45:46.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.016+0000 7f102a2a5640 1 Processor -- start 2024-03-25T17:45:46.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.016+0000 7f102a2a5640 1 -- start start 2024-03-25T17:45:46.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.016+0000 7f102a2a5640 1 librados: setting wanted keys 2024-03-25T17:45:46.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.016+0000 7f102a2a5640 1 librados: calling monclient init 2024-03-25T17:45:46.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.016+0000 7f102a2a5640 10 monclient: init 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 10 monclient(hunting): picked mon.noname-a con 0x7f10240b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 10 monclient(hunting): picked mon.noname-c con 0x7f10240b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 10 monclient(hunting): picked mon.noname-b con 0x7f102414be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1024138340 con 0x7f10240b1730 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1024137a50 con 0x7f102414be90 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1024137be0 con 0x7f10240b4d60 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:46.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.017+0000 7f102a2a5640 10 monclient(hunting): authenticate will time out at 6349.244920s 2024-03-25T17:45:46.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.025+0000 7f1028842640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f102414be90 0x7f10241e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:59190/0 (socket says 172.21.15.44:59190) 2024-03-25T17:45:46.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.025+0000 7f1028842640 1 -- 172.21.15.44:0/132753985 learned_addr learned my addr 172.21.15.44:0/132753985 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:46.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.026+0000 7f1020ff9640 10 client.?.objecter ms_handle_connect 0x7f102414be90 2024-03-25T17:45:46.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.026+0000 7f1020ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:46.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.029+0000 7f1020ff9640 10 client.?.objecter ms_handle_connect 0x7f10240b4d60 2024-03-25T17:45:46.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.029+0000 7f1020ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:46.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.031+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230195719 0 0) 0x7f1024137a50 con 0x7f102414be90 2024-03-25T17:45:46.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.031+0000 7f1020ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:45:46.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.031+0000 7f1020ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:46.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.031+0000 7f1020ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:46.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.031+0000 7f1020ff9640 10 monclient(hunting): my global_id is 4516 2024-03-25T17:45:46.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.031+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1010004eb0 con 0x7f102414be90 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047182088 0 0) 0x7f1024137be0 con 0x7f10240b4d60 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 10 monclient(hunting): my global_id is 4523 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1024137a50 con 0x7f10240b4d60 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114611490 0 0) 0x7f1010004eb0 con 0x7f102414be90 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1024137be0 con 0x7f102414be90 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f101c001fa0 con 0x7f102414be90 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:45:46.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 10 monclient(hunting): dump: 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:46.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.046+0000 7f1020ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.050+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1370512406 0 0) 0x7f1024137be0 con 0x7f102414be90 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 >> v1:172.21.15.44:6790/0 conn(0x7f10240b4d60 legacy=0x7f10241dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 >> v1:172.21.15.44:6789/0 conn(0x7f10240b1730 legacy=0x7f1024131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 1 monclient: found mon.b 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1024140380 con 0x7f102414be90 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 10 monclient: _finish_auth 0 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f1020ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 5 monclient: authenticate success, global_id 4516 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 10 monclient: _renew_subs 2024-03-25T17:45:46.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:46.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10241405d0 con 0x7f102414be90 2024-03-25T17:45:46.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:45:46.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 10 monclient: _renew_subs 2024-03-25T17:45:46.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:46.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1024144f20 con 0x7f102414be90 2024-03-25T17:45:46.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.051+0000 7f102a2a5640 1 librados: init done 2024-03-25T17:45:46.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.052+0000 7f102a2a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:46.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.052+0000 7f102a2a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:46.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.052+0000 7f102a2a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:46.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.052+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10240627b0 con 0x7f102414be90 2024-03-25T17:45:46.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.319+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f101c003c20 con 0x7f102414be90 2024-03-25T17:45:46.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.319+0000 7f1020ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:46.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.319+0000 7f1021ffb640 4 set_mon_vals no callback set 2024-03-25T17:45:46.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.320+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f101c003ff0 con 0x7f102414be90 2024-03-25T17:45:46.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.320+0000 7f1020ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:46.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.320+0000 7f1020ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:46.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.320+0000 7f1020ff9640 10 monclient: dump: 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:46.321 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:46.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.320+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f101c004210 con 0x7f102414be90 2024-03-25T17:45:46.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.321+0000 7f1023fff640 10 monclient: get_auth_request con 0x7f101006dcb0 auth_method 0 2024-03-25T17:45:46.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.321+0000 7f1020ff9640 10 client.4516.objecter ms_handle_connect 0x7f101006dcb0 2024-03-25T17:45:46.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.325+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f101c054b30 con 0x7f102414be90 2024-03-25T17:45:46.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.325+0000 7f1020ff9640 10 client.4516.objecter ms_dispatch 0x7f1024000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:45:46.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.325+0000 7f1020ff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:45:46.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.325+0000 7f1020ff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:45:46.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.325+0000 7f1020ff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T17:45:46.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.326+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f101c050640 con 0x7f102414be90 2024-03-25T17:45:46.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.326+0000 7f1020ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:46.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.326+0000 7f1020ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:45:46.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.543+0000 7f102a2a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:46.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.543+0000 7f102a2a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:46.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.543+0000 7f102a2a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:46.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.543+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1024069f40 con 0x7f102414be90 2024-03-25T17:45:46.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.607+0000 7f1020ff9640 1 -- 172.21.15.44:0/132753985 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f101c052570 con 0x7f102414be90 2024-03-25T17:45:46.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.608+0000 7f1020ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:46.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.608+0000 7f1020ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:45:46.608 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:45:46.608 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:45:46.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 10 librados: watch_flush enter 2024-03-25T17:45:46.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 10 librados: watch_flush exit 2024-03-25T17:45:46.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T17:45:46.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 10 client.4516.objecter successfully canceled tick 2024-03-25T17:45:46.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 >> v1:172.21.15.44:6800/3257043132 conn(0x7f101006dcb0 legacy=0x7f1010070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:46.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 10 monclient: shutdown 2024-03-25T17:45:46.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:46.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 >> v1:172.21.15.111:6789/0 conn(0x7f102414be90 legacy=0x7f10241e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:46.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 shutdown_connections 2024-03-25T17:45:46.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 >> 172.21.15.44:0/132753985 conn(0x7f10240617a0 msgr2=0x7f10240b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:46.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 shutdown_connections 2024-03-25T17:45:46.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 1 -- 172.21.15.44:0/132753985 wait complete. 2024-03-25T17:45:46.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:46.610+0000 7f102a2a5640 1 librados: shutdown 2024-03-25T17:45:46.623 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:45:53.430 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.545+0000 7f1f51b15640 10 monclient: get_monmap_and_config 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.545+0000 7f1f51b15640 10 monclient: build_initial_monmap 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.545+0000 7f1f51b15640 10 monclient: monmap: 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:45:53.546581+0000 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:45:53.546581+0000 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:53.546 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:53.547 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:53.547 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:45:53.547 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:45:53.547 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.546+0000 7f1f51b15640 1 Processor -- start 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.546+0000 7f1f51b15640 1 -- start start 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.546+0000 7f1f51b15640 10 monclient: init 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 10 monclient(hunting): picked mon.noname-b con 0x7f1f4c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 10 monclient(hunting): picked mon.noname-c con 0x7f1f4c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c142520 con 0x7f1f4c0a98f0 2024-03-25T17:45:53.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c0b28a0 con 0x7f1f4c1430c0 2024-03-25T17:45:53.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c0b2a30 con 0x7f1f4c13f710 2024-03-25T17:45:53.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:53.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.547+0000 7f1f51b15640 10 monclient(hunting): authenticate will time out at 6356.775235s 2024-03-25T17:45:53.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.557+0000 7f1f4affd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f1f4c13f710 0x7f1f4c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:56086/0 (socket says 172.21.15.44:56086) 2024-03-25T17:45:53.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.557+0000 7f1f4affd640 1 -- 172.21.15.44:0/3187101670 learned_addr learned my addr 172.21.15.44:0/3187101670 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:53.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.568+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033123052 0 0) 0x7f1f4c0b2a30 con 0x7f1f4c13f710 2024-03-25T17:45:53.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.568+0000 7f1f4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:45:53.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.568+0000 7f1f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:53.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.568+0000 7f1f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:53.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.568+0000 7f1f4a7fc640 10 monclient(hunting): my global_id is 4526 2024-03-25T17:45:53.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.568+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f34003520 con 0x7f1f4c13f710 2024-03-25T17:45:53.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.576+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514258802 0 0) 0x7f1f4c142520 con 0x7f1f4c0a98f0 2024-03-25T17:45:53.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.576+0000 7f1f4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:45:53.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.576+0000 7f1f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:53.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.576+0000 7f1f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:53.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.576+0000 7f1f4a7fc640 10 monclient(hunting): my global_id is 4533 2024-03-25T17:45:53.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.576+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f4c0b2a30 con 0x7f1f4c0a98f0 2024-03-25T17:45:53.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 477223519 0 0) 0x7f1f34003520 con 0x7f1f4c13f710 2024-03-25T17:45:53.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 >> v1:172.21.15.111:6789/0 conn(0x7f1f4c1430c0 legacy=0x7f1f4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 >> v1:172.21.15.44:6789/0 conn(0x7f1f4c0a98f0 legacy=0x7f1f4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 1 monclient: found mon.noname-b 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c0b44c0 con 0x7f1f4c13f710 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 10 monclient: _finish_auth 0 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1f40001690 con 0x7f1f4c13f710 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f4a7fc640 10 monclient: dump: 2024-03-25T17:45:53.580 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f51b15640 5 monclient: authenticate success, global_id 4526 2024-03-25T17:45:53.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.579+0000 7f1f51b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:45:53.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.589+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f40001870 con 0x7f1f4c13f710 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.589+0000 7f1f4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.589+0000 7f1f4a7fc640 1 -- 172.21.15.44:0/3187101670 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1f40001c80 con 0x7f1f4c13f710 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.589+0000 7f1f4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.589+0000 7f1f4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.589+0000 7f1f4a7fc640 10 monclient: dump: 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:53.590 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:53.591 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:53.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.589+0000 7f1f51b15640 10 monclient: get_monmap_and_config success 2024-03-25T17:45:53.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.589+0000 7f1f51b15640 4 set_mon_vals no callback set 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.590+0000 7f1f51b15640 10 monclient: shutdown 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.590+0000 7f1f51b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.590+0000 7f1f51b15640 1 -- 172.21.15.44:0/3187101670 >> v1:172.21.15.44:6790/0 conn(0x7f1f4c13f710 legacy=0x7f1f4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.590+0000 7f1f51b15640 1 -- 172.21.15.44:0/3187101670 shutdown_connections 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.590+0000 7f1f51b15640 1 -- 172.21.15.44:0/3187101670 >> 172.21.15.44:0/3187101670 conn(0x7f1f4c0b7ea0 msgr2=0x7f1f4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.590+0000 7f1f51b15640 1 -- 172.21.15.44:0/3187101670 shutdown_connections 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.590+0000 7f1f51b15640 1 -- 172.21.15.44:0/3187101670 wait complete. 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 10 monclient: build_initial_monmap 2024-03-25T17:45:53.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 10 monclient: monmap: 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 1 librados: starting msgr at 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 1 librados: starting objecter 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 1 Processor -- start 2024-03-25T17:45:53.594 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 1 -- start start 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 1 librados: setting wanted keys 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 1 librados: calling monclient init 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.591+0000 7f1f51b15640 10 monclient: init 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 10 monclient: _reopen_session rank -1 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 10 monclient(hunting): picked mon.noname-c con 0x7f1f4c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 10 monclient(hunting): picked mon.noname-b con 0x7f1f4c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c071010 con 0x7f1f4c0a98f0 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c06fe80 con 0x7f1f4c1430c0 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c070160 con 0x7f1f4c13f710 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 10 monclient(hunting): _renew_subs 2024-03-25T17:45:53.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.592+0000 7f1f51b15640 10 monclient(hunting): authenticate will time out at 6356.820027s 2024-03-25T17:45:53.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.600+0000 7f1f4affd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f1f4c13f710 0x7f1f4c1e32b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:56116/0 (socket says 172.21.15.44:56116) 2024-03-25T17:45:53.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.600+0000 7f1f4affd640 1 -- 172.21.15.44:0/3640084722 learned_addr learned my addr 172.21.15.44:0/3640084722 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:45:53.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.601+0000 7f1f33fff640 10 client.?.objecter ms_handle_connect 0x7f1f4c13f710 2024-03-25T17:45:53.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.602+0000 7f1f33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:53.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.613+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085961659 0 0) 0x7f1f4c070160 con 0x7f1f4c13f710 2024-03-25T17:45:53.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.613+0000 7f1f33fff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:53.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.613+0000 7f1f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:53.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.613+0000 7f1f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:53.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.613+0000 7f1f33fff640 10 monclient(hunting): my global_id is 4529 2024-03-25T17:45:53.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.613+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f38004eb0 con 0x7f1f4c13f710 2024-03-25T17:45:53.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.620+0000 7f1f33fff640 10 client.?.objecter ms_handle_connect 0x7f1f4c1430c0 2024-03-25T17:45:53.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.620+0000 7f1f33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:53.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.625+0000 7f1f33fff640 10 client.?.objecter ms_handle_connect 0x7f1f4c0a98f0 2024-03-25T17:45:53.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.625+0000 7f1f33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:45:53.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.633+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341476002 0 0) 0x7f1f4c06fe80 con 0x7f1f4c1430c0 2024-03-25T17:45:53.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.633+0000 7f1f33fff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:53.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.633+0000 7f1f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:53.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.633+0000 7f1f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:53.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.633+0000 7f1f33fff640 10 monclient(hunting): my global_id is 4519 2024-03-25T17:45:53.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.633+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f4c070160 con 0x7f1f4c1430c0 2024-03-25T17:45:53.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.636+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929017198 0 0) 0x7f1f4c071010 con 0x7f1f4c0a98f0 2024-03-25T17:45:53.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.636+0000 7f1f33fff640 10 monclient(hunting): _negotiate 2024-03-25T17:45:53.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.636+0000 7f1f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:45:53.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.636+0000 7f1f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:45:53.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.636+0000 7f1f33fff640 10 monclient(hunting): my global_id is 4536 2024-03-25T17:45:53.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.636+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f4c06fe80 con 0x7f1f4c0a98f0 2024-03-25T17:45:53.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.645+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1340948525 0 0) 0x7f1f38004eb0 con 0x7f1f4c13f710 2024-03-25T17:45:53.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.646+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f4c071010 con 0x7f1f4c13f710 2024-03-25T17:45:53.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.646+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1f40001c40 con 0x7f1f4c13f710 2024-03-25T17:45:53.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.646+0000 7f1f33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:53.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.646+0000 7f1f33fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:45:53.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.646+0000 7f1f33fff640 10 monclient(hunting): dump: 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:53.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.646+0000 7f1f33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:45:53.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.648+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3441438255 0 0) 0x7f1f4c070160 con 0x7f1f4c1430c0 2024-03-25T17:45:53.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.648+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f38004eb0 con 0x7f1f4c1430c0 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.648+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1f44001b10 con 0x7f1f4c1430c0 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.648+0000 7f1f33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.648+0000 7f1f33fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.648+0000 7f1f33fff640 10 monclient(hunting): dump: 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:53.649 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:53.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1794452824 0 0) 0x7f1f38004eb0 con 0x7f1f4c1430c0 2024-03-25T17:45:53.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 >> v1:172.21.15.44:6790/0 conn(0x7f1f4c13f710 legacy=0x7f1f4c1e32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:53.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 >> v1:172.21.15.44:6789/0 conn(0x7f1f4c0a98f0 legacy=0x7f1f4c14bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:53.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 10 monclient: _finish_hunting 0 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 1 monclient: found mon.b 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c0714a0 con 0x7f1f4c1430c0 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 10 monclient: _finish_auth 0 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 10 monclient: _check_auth_tickets 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 5 monclient: authenticate success, global_id 4519 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 10 monclient: _renew_subs 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c14e690 con 0x7f1f4c1430c0 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 10 monclient: _renew_subs 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:53.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c14c3c0 con 0x7f1f4c1430c0 2024-03-25T17:45:53.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.653+0000 7f1f51b15640 1 librados: init done 2024-03-25T17:45:53.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.654+0000 7f1f51b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:53.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.654+0000 7f1f51b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:53.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.654+0000 7f1f51b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:53.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.654+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4c0627b0 con 0x7f1f4c1430c0 2024-03-25T17:45:53.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.926+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f44002000 con 0x7f1f4c1430c0 2024-03-25T17:45:53.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.926+0000 7f1f33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:45:53.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.927+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1f44003b10 con 0x7f1f4c1430c0 2024-03-25T17:45:53.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.927+0000 7f1f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.927+0000 7f1f33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.927+0000 7f1f33fff640 10 monclient: dump: 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:45:53.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.927+0000 7f1f49ffb640 4 set_mon_vals no callback set 2024-03-25T17:45:53.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.928+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f1f44003d30 con 0x7f1f4c1430c0 2024-03-25T17:45:53.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.928+0000 7f1f4b7fe640 10 monclient: get_auth_request con 0x7f1f3806df60 auth_method 0 2024-03-25T17:45:53.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.929+0000 7f1f33fff640 10 client.4519.objecter ms_handle_connect 0x7f1f3806df60 2024-03-25T17:45:53.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.932+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f1f440537e0 con 0x7f1f4c1430c0 2024-03-25T17:45:53.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.932+0000 7f1f33fff640 10 client.4519.objecter ms_dispatch 0x7f1f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:45:53.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.932+0000 7f1f33fff640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:45:53.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.932+0000 7f1f33fff640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:45:53.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.932+0000 7f1f33fff640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-25T17:45:53.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.933+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f4404f2f0 con 0x7f1f4c1430c0 2024-03-25T17:45:53.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.933+0000 7f1f33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:45:53.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:53.933+0000 7f1f33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:45:54.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.170+0000 7f1f51b15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:54.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.171+0000 7f1f51b15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:54.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.171+0000 7f1f51b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:45:54.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.171+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f4c0711c0 con 0x7f1f4c1430c0 2024-03-25T17:45:54.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.272+0000 7f1f33fff640 1 -- 172.21.15.44:0/3640084722 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f1f4c0711c0 con 0x7f1f4c1430c0 2024-03-25T17:45:54.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.272+0000 7f1f33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:45:54.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.272+0000 7f1f33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:45:54.273 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:45:54.273 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:45:54.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 10 librados: watch_flush enter 2024-03-25T17:45:54.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 10 librados: watch_flush exit 2024-03-25T17:45:54.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-25T17:45:54.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 10 client.4519.objecter successfully canceled tick 2024-03-25T17:45:54.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 >> v1:172.21.15.44:6800/3257043132 conn(0x7f1f3806df60 legacy=0x7f1f38070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:54.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 10 monclient: shutdown 2024-03-25T17:45:54.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:45:54.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 >> v1:172.21.15.111:6789/0 conn(0x7f1f4c1430c0 legacy=0x7f1f4c1e69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:45:54.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 shutdown_connections 2024-03-25T17:45:54.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.274+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 >> 172.21.15.44:0/3640084722 conn(0x7f1f4c0b7ea0 msgr2=0x7f1f4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:45:54.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.275+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 shutdown_connections 2024-03-25T17:45:54.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.275+0000 7f1f51b15640 1 -- 172.21.15.44:0/3640084722 wait complete. 2024-03-25T17:45:54.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:45:54.275+0000 7f1f51b15640 1 librados: shutdown 2024-03-25T17:45:54.287 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:45:58.355 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:15.711 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:58.407 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:15.763 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:58.444 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:15.800 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:58.469 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:15.789 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:58.474 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:15.830 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:58.479 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:15.835 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:58.482 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:15.802 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:58.505 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:15.825 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:45:58.517 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:15.873 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:01.095 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:46:01.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.208+0000 7efe2268f640 10 monclient: get_monmap_and_config 2024-03-25T17:46:01.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.208+0000 7efe2268f640 10 monclient: build_initial_monmap 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.208+0000 7efe2268f640 10 monclient: monmap: 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:46:01.210890+0000 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:46:01.210890+0000 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:46:01.211 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:01.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.209+0000 7efe2268f640 1 Processor -- start 2024-03-25T17:46:01.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.209+0000 7efe2268f640 1 -- start start 2024-03-25T17:46:01.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.209+0000 7efe2268f640 10 monclient: init 2024-03-25T17:46:01.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 10 monclient(hunting): picked mon.noname-a con 0x7efe1c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 10 monclient(hunting): picked mon.noname-b con 0x7efe1c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 10 monclient(hunting): picked mon.noname-c con 0x7efe1c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c142520 con 0x7efe1c0a98f0 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0b28a0 con 0x7efe1c1430c0 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0b2a30 con 0x7efe1c13f710 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.210+0000 7efe2268f640 10 monclient(hunting): authenticate will time out at 6364.439278s 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.211+0000 7efe20c2c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efe1c1430c0 0x7efe1c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:34946/0 (socket says 172.21.15.44:34946) 2024-03-25T17:46:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.211+0000 7efe20c2c640 1 -- 172.21.15.44:0/966230830 learned_addr learned my addr 172.21.15.44:0/966230830 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:01.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.273+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188592696 0 0) 0x7efe1c0b2a30 con 0x7efe1c13f710 2024-03-25T17:46:01.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.273+0000 7efe1affd640 10 monclient(hunting): _negotiate 2024-03-25T17:46:01.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.273+0000 7efe1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:01.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.273+0000 7efe1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:01.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.273+0000 7efe1affd640 10 monclient(hunting): my global_id is 4532 2024-03-25T17:46:01.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.273+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe04003520 con 0x7efe1c13f710 2024-03-25T17:46:01.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.292+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213358481 0 0) 0x7efe1c142520 con 0x7efe1c0a98f0 2024-03-25T17:46:01.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.292+0000 7efe1affd640 10 monclient(hunting): _negotiate 2024-03-25T17:46:01.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.292+0000 7efe1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:01.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.292+0000 7efe1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:01.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.292+0000 7efe1affd640 10 monclient(hunting): my global_id is 4539 2024-03-25T17:46:01.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.292+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1c0b2a30 con 0x7efe1c0a98f0 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3466540840 0 0) 0x7efe04003520 con 0x7efe1c13f710 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 >> v1:172.21.15.111:6789/0 conn(0x7efe1c1430c0 legacy=0x7efe1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 >> v1:172.21.15.44:6789/0 conn(0x7efe1c0a98f0 legacy=0x7efe1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 10 monclient: _finish_hunting 0 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 1 monclient: found mon.noname-b 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c0b44c0 con 0x7efe1c13f710 2024-03-25T17:46:01.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 10 monclient: _finish_auth 0 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 10 monclient: _check_auth_tickets 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efe10001690 con 0x7efe1c13f710 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe2268f640 5 monclient: authenticate success, global_id 4532 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe2268f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe1affd640 10 monclient: dump: 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:01.301 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:01.302 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:01.302 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:01.302 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:01.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.298+0000 7efe2268f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.303+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe100017f0 con 0x7efe1c13f710 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.303+0000 7efe1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.303+0000 7efe1affd640 1 -- 172.21.15.44:0/966230830 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efe10001c00 con 0x7efe1c13f710 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.303+0000 7efe2268f640 10 monclient: get_monmap_and_config success 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.303+0000 7efe2268f640 4 set_mon_vals no callback set 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.303+0000 7efe1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.303+0000 7efe1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.303+0000 7efe1affd640 10 monclient: dump: 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.305 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.306 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:01.306 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:01.306 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:01.306 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:01.306 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:01.306 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:01.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.304+0000 7efe2268f640 10 monclient: shutdown 2024-03-25T17:46:01.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.304+0000 7efe2268f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:01.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.304+0000 7efe2268f640 1 -- 172.21.15.44:0/966230830 >> v1:172.21.15.44:6790/0 conn(0x7efe1c13f710 legacy=0x7efe1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:01.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.304+0000 7efe2268f640 1 -- 172.21.15.44:0/966230830 shutdown_connections 2024-03-25T17:46:01.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.304+0000 7efe2268f640 1 -- 172.21.15.44:0/966230830 >> 172.21.15.44:0/966230830 conn(0x7efe1c0b7ea0 msgr2=0x7efe1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.304+0000 7efe2268f640 1 -- 172.21.15.44:0/966230830 shutdown_connections 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.304+0000 7efe2268f640 1 -- 172.21.15.44:0/966230830 wait complete. 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.305+0000 7efe2268f640 10 monclient: build_initial_monmap 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.305+0000 7efe2268f640 10 monclient: monmap: 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:46:01.308 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:01.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.305+0000 7efe2268f640 1 librados: starting msgr at 2024-03-25T17:46:01.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.305+0000 7efe2268f640 1 librados: starting objecter 2024-03-25T17:46:01.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.306+0000 7efe2268f640 1 Processor -- start 2024-03-25T17:46:01.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.306+0000 7efe2268f640 1 -- start start 2024-03-25T17:46:01.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.306+0000 7efe2268f640 1 librados: setting wanted keys 2024-03-25T17:46:01.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.306+0000 7efe2268f640 1 librados: calling monclient init 2024-03-25T17:46:01.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.306+0000 7efe2268f640 10 monclient: init 2024-03-25T17:46:01.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:01.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:46:01.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 10 monclient(hunting): picked mon.noname-b con 0x7efe1c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:01.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 10 monclient(hunting): picked mon.noname-c con 0x7efe1c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:01.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 10 monclient(hunting): picked mon.noname-a con 0x7efe1c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:01.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0aba00 con 0x7efe1c1430c0 2024-03-25T17:46:01.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c1d7490 con 0x7efe1c0a98f0 2024-03-25T17:46:01.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c1d8360 con 0x7efe1c13f710 2024-03-25T17:46:01.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:01.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.307+0000 7efe2268f640 10 monclient(hunting): authenticate will time out at 6364.536055s 2024-03-25T17:46:01.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.319+0000 7efe20c2c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7efe1c1430c0 0x7efe1c133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55354/0 (socket says 172.21.15.44:55354) 2024-03-25T17:46:01.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.319+0000 7efe20c2c640 1 -- 172.21.15.44:0/82570465 learned_addr learned my addr 172.21.15.44:0/82570465 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:01.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.323+0000 7efe18ff9640 10 client.?.objecter ms_handle_connect 0x7efe1c13f710 2024-03-25T17:46:01.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.323+0000 7efe18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:01.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.325+0000 7efe18ff9640 10 client.?.objecter ms_handle_connect 0x7efe1c1430c0 2024-03-25T17:46:01.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.325+0000 7efe18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:01.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.330+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042517650 0 0) 0x7efe1c1d8360 con 0x7efe1c13f710 2024-03-25T17:46:01.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.331+0000 7efe18ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:46:01.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.331+0000 7efe18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:01.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.331+0000 7efe18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:01.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.331+0000 7efe18ff9640 10 monclient(hunting): my global_id is 4535 2024-03-25T17:46:01.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.331+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe08004eb0 con 0x7efe1c13f710 2024-03-25T17:46:01.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.337+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384966023 0 0) 0x7efe1c0aba00 con 0x7efe1c1430c0 2024-03-25T17:46:01.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.337+0000 7efe18ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:46:01.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.337+0000 7efe18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:01.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.337+0000 7efe18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:01.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.337+0000 7efe18ff9640 10 monclient(hunting): my global_id is 4542 2024-03-25T17:46:01.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.337+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1c1d8360 con 0x7efe1c1430c0 2024-03-25T17:46:01.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.351+0000 7efe18ff9640 10 client.?.objecter ms_handle_connect 0x7efe1c0a98f0 2024-03-25T17:46:01.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.351+0000 7efe18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:01.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.356+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2910159147 0 0) 0x7efe08004eb0 con 0x7efe1c13f710 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.356+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe1c0aba00 con 0x7efe1c13f710 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.356+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efe10001bc0 con 0x7efe1c13f710 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.356+0000 7efe18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.356+0000 7efe18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.356+0000 7efe18ff9640 10 monclient(hunting): dump: 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:01.358 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:01.359 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:01.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.356+0000 7efe18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:46:01.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990955696 0 0) 0x7efe1c1d7490 con 0x7efe1c0a98f0 2024-03-25T17:46:01.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:46:01.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:01.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:01.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 10 monclient(hunting): my global_id is 4525 2024-03-25T17:46:01.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe08004eb0 con 0x7efe1c0a98f0 2024-03-25T17:46:01.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1993339666 0 0) 0x7efe1c1d8360 con 0x7efe1c1430c0 2024-03-25T17:46:01.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe1c1d7490 con 0x7efe1c1430c0 2024-03-25T17:46:01.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efe14001b90 con 0x7efe1c1430c0 2024-03-25T17:46:01.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:01.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:46:01.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.359+0000 7efe18ff9640 10 monclient(hunting): dump: 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:01.362 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:01.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4096505761 0 0) 0x7efe1c1d7490 con 0x7efe1c1430c0 2024-03-25T17:46:01.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 >> v1:172.21.15.44:6790/0 conn(0x7efe1c13f710 legacy=0x7efe1c1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:01.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 >> v1:172.21.15.111:6789/0 conn(0x7efe1c0a98f0 legacy=0x7efe1c1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:01.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:46:01.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 1 monclient: found mon.a 2024-03-25T17:46:01.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:01.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c0abc30 con 0x7efe1c1430c0 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 10 monclient: _finish_auth 0 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 5 monclient: authenticate success, global_id 4542 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 10 monclient: _renew_subs 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1c137940 con 0x7efe1c1430c0 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 10 monclient: _renew_subs 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1c135220 con 0x7efe1c1430c0 2024-03-25T17:46:01.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.369+0000 7efe2268f640 1 librados: init done 2024-03-25T17:46:01.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.370+0000 7efe2268f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:01.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.370+0000 7efe2268f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:01.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.370+0000 7efe2268f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:01.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.370+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1c0627b0 con 0x7efe1c1430c0 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.676+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe14001fc0 con 0x7efe1c1430c0 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.676+0000 7efe18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.676+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efe14004d00 con 0x7efe1c1430c0 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.676+0000 7efe18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.676+0000 7efe18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.676+0000 7efe18ff9640 10 monclient: dump: 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.678 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:01.679 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:01.679 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:01.679 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:01.679 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:01.679 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:01.679 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:01.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.676+0000 7efe1a7fc640 4 set_mon_vals no callback set 2024-03-25T17:46:01.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.677+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7efe14004f20 con 0x7efe1c1430c0 2024-03-25T17:46:01.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.678+0000 7efe1bfff640 10 monclient: get_auth_request con 0x7efe0806df50 auth_method 0 2024-03-25T17:46:01.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.678+0000 7efe18ff9640 10 client.4542.objecter ms_handle_connect 0x7efe0806df50 2024-03-25T17:46:01.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.688+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7efe14054a00 con 0x7efe1c1430c0 2024-03-25T17:46:01.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.688+0000 7efe18ff9640 10 client.4542.objecter ms_dispatch 0x7efe1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:46:01.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.688+0000 7efe18ff9640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:46:01.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.688+0000 7efe18ff9640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:46:01.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.688+0000 7efe18ff9640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-25T17:46:01.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.688+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe14050510 con 0x7efe1c1430c0 2024-03-25T17:46:01.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.688+0000 7efe18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:01.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.688+0000 7efe18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:46:01.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.943+0000 7efe2268f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:01.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.943+0000 7efe2268f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:01.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.943+0000 7efe2268f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:01.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.943+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe1c069f40 con 0x7efe1c1430c0 2024-03-25T17:46:01.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.976+0000 7efe18ff9640 1 -- 172.21.15.44:0/82570465 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7efe14052440 con 0x7efe1c1430c0 2024-03-25T17:46:01.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.977+0000 7efe18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:01.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.977+0000 7efe18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:46:01.979 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:46:01.979 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:46:01.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.978+0000 7efe2268f640 10 librados: watch_flush enter 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 10 librados: watch_flush exit 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 10 client.4542.objecter successfully canceled tick 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 >> v1:172.21.15.44:6800/3257043132 conn(0x7efe0806df50 legacy=0x7efe08070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 10 monclient: shutdown 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 >> v1:172.21.15.44:6789/0 conn(0x7efe1c1430c0 legacy=0x7efe1c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 shutdown_connections 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 >> 172.21.15.44:0/82570465 conn(0x7efe1c0b7ea0 msgr2=0x7efe1c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 shutdown_connections 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 1 -- 172.21.15.44:0/82570465 wait complete. 2024-03-25T17:46:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:01.979+0000 7efe2268f640 1 librados: shutdown 2024-03-25T17:46:01.992 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:46:08.799 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.914+0000 7f95d3a35640 10 monclient: get_monmap_and_config 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.914+0000 7f95d3a35640 10 monclient: build_initial_monmap 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.914+0000 7f95d3a35640 10 monclient: monmap: 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:46:08.915425+0000 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:46:08.915425+0000 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:08.915 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:46:08.916 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:46:08.916 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:08.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.915+0000 7f95d3a35640 1 Processor -- start 2024-03-25T17:46:08.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.915+0000 7f95d3a35640 1 -- start start 2024-03-25T17:46:08.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.915+0000 7f95d3a35640 10 monclient: init 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 10 monclient(hunting): picked mon.noname-a con 0x7f95cc0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 10 monclient(hunting): picked mon.noname-b con 0x7f95cc13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 10 monclient(hunting): picked mon.noname-c con 0x7f95cc1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc142520 con 0x7f95cc0a98f0 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc0b28a0 con 0x7f95cc1430c0 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc0b2a30 con 0x7f95cc13f710 2024-03-25T17:46:08.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:08.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.916+0000 7f95d3a35640 10 monclient(hunting): authenticate will time out at 6372.144392s 2024-03-25T17:46:08.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.924+0000 7f95d1fd2640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f95cc1430c0 0x7f95cc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:39840/0 (socket says 172.21.15.44:39840) 2024-03-25T17:46:08.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.924+0000 7f95d1fd2640 1 -- 172.21.15.44:0/3780204484 learned_addr learned my addr 172.21.15.44:0/3780204484 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:08.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.936+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882431754 0 0) 0x7f95cc0b28a0 con 0x7f95cc1430c0 2024-03-25T17:46:08.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.936+0000 7f95c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:46:08.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.936+0000 7f95c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:08.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.936+0000 7f95c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:08.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.936+0000 7f95c3fff640 10 monclient(hunting): my global_id is 4528 2024-03-25T17:46:08.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.936+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b8003520 con 0x7f95cc1430c0 2024-03-25T17:46:08.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.940+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889412503 0 0) 0x7f95cc142520 con 0x7f95cc0a98f0 2024-03-25T17:46:08.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.940+0000 7f95c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:46:08.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.940+0000 7f95c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:08.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.940+0000 7f95c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:08.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.940+0000 7f95c3fff640 10 monclient(hunting): my global_id is 4545 2024-03-25T17:46:08.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.940+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95cc0b28a0 con 0x7f95cc0a98f0 2024-03-25T17:46:08.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89602699 0 0) 0x7f95b8003520 con 0x7f95cc1430c0 2024-03-25T17:46:08.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 >> v1:172.21.15.44:6790/0 conn(0x7f95cc13f710 legacy=0x7f95cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:08.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 >> v1:172.21.15.44:6789/0 conn(0x7f95cc0a98f0 legacy=0x7f95cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:08.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:46:08.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 1 monclient: found mon.noname-c 2024-03-25T17:46:08.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc0b44c0 con 0x7f95cc1430c0 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 10 monclient: _finish_auth 0 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f95c4001a10 con 0x7f95cc1430c0 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95d3a35640 5 monclient: authenticate success, global_id 4528 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95d3a35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95c3fff640 10 monclient: dump: 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:08.943 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:08.944 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:08.944 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:08.944 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:08.944 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:08.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.941+0000 7f95d3a35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:08.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.947+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95c4001b70 con 0x7f95cc1430c0 2024-03-25T17:46:08.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.947+0000 7f95c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.947+0000 7f95c3fff640 1 -- 172.21.15.44:0/3780204484 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f95c4001f80 con 0x7f95cc1430c0 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.947+0000 7f95d3a35640 10 monclient: get_monmap_and_config success 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.947+0000 7f95d3a35640 4 set_mon_vals no callback set 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.947+0000 7f95c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.947+0000 7f95c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.947+0000 7f95c3fff640 10 monclient: dump: 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:08.948 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:08.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.948+0000 7f95d3a35640 10 monclient: shutdown 2024-03-25T17:46:08.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.948+0000 7f95d3a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.948+0000 7f95d3a35640 1 -- 172.21.15.44:0/3780204484 >> v1:172.21.15.111:6789/0 conn(0x7f95cc1430c0 legacy=0x7f95cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.948+0000 7f95d3a35640 1 -- 172.21.15.44:0/3780204484 shutdown_connections 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.948+0000 7f95d3a35640 1 -- 172.21.15.44:0/3780204484 >> 172.21.15.44:0/3780204484 conn(0x7f95cc0b7ea0 msgr2=0x7f95cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.948+0000 7f95d3a35640 1 -- 172.21.15.44:0/3780204484 shutdown_connections 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.948+0000 7f95d3a35640 1 -- 172.21.15.44:0/3780204484 wait complete. 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.949+0000 7f95d3a35640 10 monclient: build_initial_monmap 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.949+0000 7f95d3a35640 10 monclient: monmap: 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:46:08.950 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:08.951 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:46:08.951 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:46:08.951 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:08.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.949+0000 7f95d3a35640 1 librados: starting msgr at 2024-03-25T17:46:08.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.949+0000 7f95d3a35640 1 librados: starting objecter 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.949+0000 7f95d3a35640 1 Processor -- start 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 1 -- start start 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 1 librados: setting wanted keys 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 1 librados: calling monclient init 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 10 monclient: init 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 10 monclient(hunting): picked mon.noname-a con 0x7f95cc0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 10 monclient(hunting): picked mon.noname-c con 0x7f95cc13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 10 monclient(hunting): picked mon.noname-b con 0x7f95cc1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc1d7590 con 0x7f95cc0a98f0 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc1d51f0 con 0x7f95cc1430c0 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc1d8410 con 0x7f95cc13f710 2024-03-25T17:46:08.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:08.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.950+0000 7f95d3a35640 10 monclient(hunting): authenticate will time out at 6372.178369s 2024-03-25T17:46:08.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.957+0000 7f95d1fd2640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f95cc1430c0 0x7f95cc1e1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:39892/0 (socket says 172.21.15.44:39892) 2024-03-25T17:46:08.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.957+0000 7f95d1fd2640 1 -- 172.21.15.44:0/3783542994 learned_addr learned my addr 172.21.15.44:0/3783542994 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:08.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.961+0000 7f95c1ffb640 10 client.?.objecter ms_handle_connect 0x7f95cc1430c0 2024-03-25T17:46:08.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.961+0000 7f95c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:08.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.967+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070061694 0 0) 0x7f95cc1d51f0 con 0x7f95cc1430c0 2024-03-25T17:46:08.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.967+0000 7f95c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:46:08.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.967+0000 7f95c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:08.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.967+0000 7f95c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:08.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.967+0000 7f95c1ffb640 10 monclient(hunting): my global_id is 4531 2024-03-25T17:46:08.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.967+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b0004eb0 con 0x7f95cc1430c0 2024-03-25T17:46:08.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.970+0000 7f95c1ffb640 10 client.?.objecter ms_handle_connect 0x7f95cc0a98f0 2024-03-25T17:46:08.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.970+0000 7f95c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:08.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.975+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506682896 0 0) 0x7f95cc1d7590 con 0x7f95cc0a98f0 2024-03-25T17:46:08.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.975+0000 7f95c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:46:08.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.975+0000 7f95c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:08.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.975+0000 7f95c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:08.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.975+0000 7f95c1ffb640 10 monclient(hunting): my global_id is 4548 2024-03-25T17:46:08.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.975+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95cc1d51f0 con 0x7f95cc0a98f0 2024-03-25T17:46:08.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.977+0000 7f95c1ffb640 10 client.?.objecter ms_handle_connect 0x7f95cc13f710 2024-03-25T17:46:08.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.977+0000 7f95c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.983+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3599936909 0 0) 0x7f95b0004eb0 con 0x7f95cc1430c0 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.983+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95cc1d7590 con 0x7f95cc1430c0 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.983+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f95c4001ec0 con 0x7f95cc1430c0 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.983+0000 7f95c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.983+0000 7f95c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.983+0000 7f95c1ffb640 10 monclient(hunting): dump: 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:08.984 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:08.985 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:08.985 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:08.985 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:08.985 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:08.985 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:08.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.983+0000 7f95c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:46:08.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.988+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250453398 0 0) 0x7f95cc1d8410 con 0x7f95cc13f710 2024-03-25T17:46:08.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.988+0000 7f95c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:46:08.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.988+0000 7f95c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:08.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.988+0000 7f95c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:08.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.988+0000 7f95c1ffb640 10 monclient(hunting): my global_id is 4541 2024-03-25T17:46:08.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.988+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b0004eb0 con 0x7f95cc13f710 2024-03-25T17:46:08.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.990+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3530351456 0 0) 0x7f95cc1d51f0 con 0x7f95cc0a98f0 2024-03-25T17:46:08.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.990+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95cc1d8410 con 0x7f95cc0a98f0 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.990+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f95bc001930 con 0x7f95cc0a98f0 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.990+0000 7f95c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.990+0000 7f95c1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.990+0000 7f95c1ffb640 10 monclient(hunting): dump: 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:08.991 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:08.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 273289984 0 0) 0x7f95cc1d7590 con 0x7f95cc1430c0 2024-03-25T17:46:08.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 >> v1:172.21.15.44:6790/0 conn(0x7f95cc13f710 legacy=0x7f95cc1385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:08.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 >> v1:172.21.15.44:6789/0 conn(0x7f95cc0a98f0 legacy=0x7f95cc1d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:08.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:46:08.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 1 monclient: found mon.b 2024-03-25T17:46:08.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:08.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:08.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc1d77e0 con 0x7f95cc1430c0 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 5 monclient: authenticate success, global_id 4531 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 10 monclient: _renew_subs 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95cc138ce0 con 0x7f95cc1430c0 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 10 monclient: _renew_subs 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95cc137ff0 con 0x7f95cc1430c0 2024-03-25T17:46:08.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.993+0000 7f95d3a35640 1 librados: init done 2024-03-25T17:46:08.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.995+0000 7f95d3a35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:08.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.995+0000 7f95d3a35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:08.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.995+0000 7f95d3a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:08.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:08.995+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95cc0627b0 con 0x7f95cc1430c0 2024-03-25T17:46:09.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.264+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95c4004e10 con 0x7f95cc1430c0 2024-03-25T17:46:09.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.264+0000 7f95c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:09.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.264+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f95c4002100 con 0x7f95cc1430c0 2024-03-25T17:46:09.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.264+0000 7f95c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:09.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.264+0000 7f95c37fe640 4 set_mon_vals no callback set 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.264+0000 7f95c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.264+0000 7f95c1ffb640 10 monclient: dump: 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:09.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.265+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f95c401af90 con 0x7f95cc1430c0 2024-03-25T17:46:09.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.265+0000 7f95d17d1640 10 monclient: get_auth_request con 0x7f95b006dfb0 auth_method 0 2024-03-25T17:46:09.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.266+0000 7f95c1ffb640 10 client.4531.objecter ms_handle_connect 0x7f95b006dfb0 2024-03-25T17:46:09.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.271+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f95c4054b90 con 0x7f95cc1430c0 2024-03-25T17:46:09.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.271+0000 7f95c1ffb640 10 client.4531.objecter ms_dispatch 0x7f95cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:46:09.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.271+0000 7f95c1ffb640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:46:09.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.271+0000 7f95c1ffb640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:46:09.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.271+0000 7f95c1ffb640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T17:46:09.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.271+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95c40504d0 con 0x7f95cc1430c0 2024-03-25T17:46:09.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.271+0000 7f95c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:09.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.271+0000 7f95c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:46:09.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.492+0000 7f95d3a35640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:09.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.492+0000 7f95d3a35640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:09.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.492+0000 7f95d3a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:09.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.492+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95cc1d7970 con 0x7f95cc1430c0 2024-03-25T17:46:09.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.516+0000 7f95c1ffb640 1 -- 172.21.15.44:0/3783542994 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f95c40525d0 con 0x7f95cc1430c0 2024-03-25T17:46:09.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.516+0000 7f95c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:09.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.516+0000 7f95c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:46:09.516 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:46:09.517 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 10 librados: watch_flush enter 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 10 librados: watch_flush exit 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 10 client.4531.objecter successfully canceled tick 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 >> v1:172.21.15.44:6800/3257043132 conn(0x7f95b006dfb0 legacy=0x7f95b0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 10 monclient: shutdown 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 >> v1:172.21.15.111:6789/0 conn(0x7f95cc1430c0 legacy=0x7f95cc1e1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 shutdown_connections 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 >> 172.21.15.44:0/3783542994 conn(0x7f95cc0b7ea0 msgr2=0x7f95cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 shutdown_connections 2024-03-25T17:46:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 1 -- 172.21.15.44:0/3783542994 wait complete. 2024-03-25T17:46:09.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:09.518+0000 7f95d3a35640 1 librados: shutdown 2024-03-25T17:46:09.530 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:46:16.336 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.450+0000 7f7fc497a640 10 monclient: get_monmap_and_config 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.450+0000 7f7fc497a640 10 monclient: build_initial_monmap 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.450+0000 7f7fc497a640 10 monclient: monmap: 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:46:16.451402+0000 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:46:16.451402+0000 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:46:16.451 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:46:16.452 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.451+0000 7f7fc497a640 1 Processor -- start 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.451+0000 7f7fc497a640 1 -- start start 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.451+0000 7f7fc497a640 10 monclient: init 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.451+0000 7f7fc497a640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.451+0000 7f7fc497a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.451+0000 7f7fc497a640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.451+0000 7f7fc497a640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.452+0000 7f7fc497a640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.452+0000 7f7fc497a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc0142520 con 0x7f7fc013f710 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.452+0000 7f7fc497a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc00b28a0 con 0x7f7fc00a98f0 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.452+0000 7f7fc497a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc00b2a30 con 0x7f7fc01430c0 2024-03-25T17:46:16.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.452+0000 7f7fc497a640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:16.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.452+0000 7f7fc497a640 10 monclient(hunting): authenticate will time out at 6379.679858s 2024-03-25T17:46:16.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.458+0000 7f7fbed9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f7fc01430c0 0x7f7fc014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:46208/0 (socket says 172.21.15.44:46208) 2024-03-25T17:46:16.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.458+0000 7f7fbed9d640 1 -- 172.21.15.44:0/3624459020 learned_addr learned my addr 172.21.15.44:0/3624459020 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:16.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.471+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370362896 0 0) 0x7f7fc00b2a30 con 0x7f7fc01430c0 2024-03-25T17:46:16.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.471+0000 7f7fbdd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:46:16.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.471+0000 7f7fbdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:16.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.471+0000 7f7fbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:16.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.471+0000 7f7fbdd9b640 10 monclient(hunting): my global_id is 4544 2024-03-25T17:46:16.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.471+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fac003520 con 0x7f7fc01430c0 2024-03-25T17:46:16.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.480+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941049299 0 0) 0x7f7fc0142520 con 0x7f7fc013f710 2024-03-25T17:46:16.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.480+0000 7f7fbdd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:46:16.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.480+0000 7f7fbdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:16.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.480+0000 7f7fbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:16.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.480+0000 7f7fbdd9b640 10 monclient(hunting): my global_id is 4551 2024-03-25T17:46:16.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.480+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc00b2a30 con 0x7f7fc013f710 2024-03-25T17:46:16.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.484+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368095707 0 0) 0x7f7fc00b28a0 con 0x7f7fc00a98f0 2024-03-25T17:46:16.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.484+0000 7f7fbdd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:46:16.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.484+0000 7f7fbdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:16.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.484+0000 7f7fbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:16.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.484+0000 7f7fbdd9b640 10 monclient(hunting): my global_id is 4534 2024-03-25T17:46:16.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.484+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc0142520 con 0x7f7fc00a98f0 2024-03-25T17:46:16.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664015223 0 0) 0x7f7fac003520 con 0x7f7fc01430c0 2024-03-25T17:46:16.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 >> v1:172.21.15.111:6789/0 conn(0x7f7fc00a98f0 legacy=0x7f7fc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:16.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 >> v1:172.21.15.44:6789/0 conn(0x7f7fc013f710 legacy=0x7f7fc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 10 monclient: _finish_hunting 0 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 1 monclient: found mon.noname-b 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc00b44c0 con 0x7f7fc01430c0 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 10 monclient: _finish_auth 0 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 10 monclient: _check_auth_tickets 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fb8001a10 con 0x7f7fc01430c0 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fc497a640 5 monclient: authenticate success, global_id 4544 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fc497a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:46:16.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fbdd9b640 10 monclient: dump: 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:16.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.492+0000 7f7fc497a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:16.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.516+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fb8001b70 con 0x7f7fc01430c0 2024-03-25T17:46:16.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.516+0000 7f7fbdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:16.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.516+0000 7f7fbdd9b640 1 -- 172.21.15.44:0/3624459020 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fb8001f80 con 0x7f7fc01430c0 2024-03-25T17:46:16.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.516+0000 7f7fbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:16.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.516+0000 7f7fbdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.516+0000 7f7fbdd9b640 10 monclient: dump: 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.516+0000 7f7fc497a640 10 monclient: get_monmap_and_config success 2024-03-25T17:46:16.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.516+0000 7f7fc497a640 4 set_mon_vals no callback set 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.517+0000 7f7fc497a640 10 monclient: shutdown 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.517+0000 7f7fc497a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.517+0000 7f7fc497a640 1 -- 172.21.15.44:0/3624459020 >> v1:172.21.15.44:6790/0 conn(0x7f7fc01430c0 legacy=0x7f7fc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.517+0000 7f7fc497a640 1 -- 172.21.15.44:0/3624459020 shutdown_connections 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.517+0000 7f7fc497a640 1 -- 172.21.15.44:0/3624459020 >> 172.21.15.44:0/3624459020 conn(0x7f7fc00b7ea0 msgr2=0x7f7fc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.517+0000 7f7fc497a640 1 -- 172.21.15.44:0/3624459020 shutdown_connections 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.517+0000 7f7fc497a640 1 -- 172.21.15.44:0/3624459020 wait complete. 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 10 monclient: build_initial_monmap 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 10 monclient: monmap: 2024-03-25T17:46:16.519 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 1 librados: starting msgr at 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 1 librados: starting objecter 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 1 Processor -- start 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 1 -- start start 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 1 librados: setting wanted keys 2024-03-25T17:46:16.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 1 librados: calling monclient init 2024-03-25T17:46:16.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.518+0000 7f7fc497a640 10 monclient: init 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc013f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc01430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc006edb0 con 0x7f7fc01430c0 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc006f090 con 0x7f7fc00a98f0 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc006f370 con 0x7f7fc013f710 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:16.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.519+0000 7f7fc497a640 10 monclient(hunting): authenticate will time out at 6379.747403s 2024-03-25T17:46:16.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.527+0000 7f7fb7fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f7fc013f710 0x7f7fc006fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:46260/0 (socket says 172.21.15.44:46260) 2024-03-25T17:46:16.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.527+0000 7f7fb7fff640 1 -- 172.21.15.44:0/1170275089 learned_addr learned my addr 172.21.15.44:0/1170275089 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:16.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.529+0000 7f7fb6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fc013f710 2024-03-25T17:46:16.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.530+0000 7f7fb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:16.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.534+0000 7f7fb6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fc00a98f0 2024-03-25T17:46:16.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.534+0000 7f7fb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:16.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.542+0000 7f7fb6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fc01430c0 2024-03-25T17:46:16.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.542+0000 7f7fb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:16.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.546+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754041124 0 0) 0x7f7fc006f090 con 0x7f7fc00a98f0 2024-03-25T17:46:16.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.546+0000 7f7fb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:46:16.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.546+0000 7f7fb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:16.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.546+0000 7f7fb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:16.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.546+0000 7f7fb6ffd640 10 monclient(hunting): my global_id is 4537 2024-03-25T17:46:16.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.546+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa0004eb0 con 0x7f7fc00a98f0 2024-03-25T17:46:16.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.556+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843528807 0 0) 0x7f7fc006edb0 con 0x7f7fc01430c0 2024-03-25T17:46:16.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.556+0000 7f7fb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:46:16.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.556+0000 7f7fb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:16.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.556+0000 7f7fb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:16.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.556+0000 7f7fb6ffd640 10 monclient(hunting): my global_id is 4554 2024-03-25T17:46:16.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.556+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc006f090 con 0x7f7fc01430c0 2024-03-25T17:46:16.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.557+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565907032 0 0) 0x7f7fc006f370 con 0x7f7fc013f710 2024-03-25T17:46:16.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.557+0000 7f7fb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:46:16.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.557+0000 7f7fb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:16.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.557+0000 7f7fb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:16.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.557+0000 7f7fb6ffd640 10 monclient(hunting): my global_id is 4547 2024-03-25T17:46:16.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.557+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc006edb0 con 0x7f7fc013f710 2024-03-25T17:46:16.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.570+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3272204466 0 0) 0x7f7fa0004eb0 con 0x7f7fc00a98f0 2024-03-25T17:46:16.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.570+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc006f370 con 0x7f7fc00a98f0 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.570+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fb00018f0 con 0x7f7fc00a98f0 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.570+0000 7f7fb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.570+0000 7f7fb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.570+0000 7f7fb6ffd640 10 monclient(hunting): dump: 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:16.571 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:16.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.570+0000 7f7fb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:46:16.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.571+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 774742767 0 0) 0x7f7fc006f090 con 0x7f7fc01430c0 2024-03-25T17:46:16.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.571+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fa0004eb0 con 0x7f7fc01430c0 2024-03-25T17:46:16.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.571+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fb8001e80 con 0x7f7fc01430c0 2024-03-25T17:46:16.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.571+0000 7f7fb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.571+0000 7f7fb6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.571+0000 7f7fb6ffd640 10 monclient(hunting): dump: 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:16.573 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:16.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.582+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2230610189 0 0) 0x7f7fc006edb0 con 0x7f7fc013f710 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.582+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc006f090 con 0x7f7fc013f710 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.582+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fa8001930 con 0x7f7fc013f710 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.582+0000 7f7fb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.582+0000 7f7fb6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.582+0000 7f7fb6ffd640 10 monclient(hunting): dump: 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:16.583 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:16.584 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:16.584 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 598132002 0 0) 0x7f7fc006f370 con 0x7f7fc00a98f0 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 >> v1:172.21.15.44:6790/0 conn(0x7f7fc013f710 legacy=0x7f7fc006fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 >> v1:172.21.15.44:6789/0 conn(0x7f7fc01430c0 legacy=0x7f7fc01e6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 1 monclient: found mon.b 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc014e6a0 con 0x7f7fc00a98f0 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:46:16.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 5 monclient: authenticate success, global_id 4537 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 10 monclient: _renew_subs 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc014e980 con 0x7f7fc00a98f0 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 10 monclient: _renew_subs 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc006e170 con 0x7f7fc00a98f0 2024-03-25T17:46:16.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.590+0000 7f7fc497a640 1 librados: init done 2024-03-25T17:46:16.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.591+0000 7f7fc497a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:16.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.591+0000 7f7fc497a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:16.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.591+0000 7f7fc497a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:16.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.591+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc00627b0 con 0x7f7fc00a98f0 2024-03-25T17:46:16.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.877+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fb0003610 con 0x7f7fc00a98f0 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.877+0000 7f7fb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.877+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fb0003920 con 0x7f7fc00a98f0 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.877+0000 7f7fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.877+0000 7f7fb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.877+0000 7f7fb6ffd640 10 monclient: dump: 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:16.878 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:16.879 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:16.879 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:16.879 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:16.879 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:16.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.877+0000 7f7fbd59a640 4 set_mon_vals no callback set 2024-03-25T17:46:16.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.878+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f7fb0019930 con 0x7f7fc00a98f0 2024-03-25T17:46:16.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.879+0000 7f7fb7fff640 10 monclient: get_auth_request con 0x7f7fa006e0f0 auth_method 0 2024-03-25T17:46:16.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.879+0000 7f7fb6ffd640 10 client.4537.objecter ms_handle_connect 0x7f7fa006e0f0 2024-03-25T17:46:16.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.883+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f7fb0053630 con 0x7f7fc00a98f0 2024-03-25T17:46:16.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.883+0000 7f7fb6ffd640 10 client.4537.objecter ms_dispatch 0x7f7fc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:46:16.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.883+0000 7f7fb6ffd640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:46:16.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.883+0000 7f7fb6ffd640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:46:16.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.883+0000 7f7fb6ffd640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-25T17:46:16.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.883+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fb004ef70 con 0x7f7fc00a98f0 2024-03-25T17:46:16.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.883+0000 7f7fb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:16.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:16.883+0000 7f7fb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:46:17.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.137+0000 7f7fc497a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:17.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.137+0000 7f7fc497a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:17.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.137+0000 7f7fc497a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:17.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.137+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fc013e230 con 0x7f7fc00a98f0 2024-03-25T17:46:17.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.213+0000 7f7fb6ffd640 1 -- 172.21.15.44:0/1170275089 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f7fb0051070 con 0x7f7fc00a98f0 2024-03-25T17:46:17.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.213+0000 7f7fb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:17.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.213+0000 7f7fb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:46:17.214 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:46:17.214 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 10 librados: watch_flush enter 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 10 librados: watch_flush exit 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 10 client.4537.objecter successfully canceled tick 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 >> v1:172.21.15.44:6800/3257043132 conn(0x7f7fa006e0f0 legacy=0x7f7fa00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 10 monclient: shutdown 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 >> v1:172.21.15.111:6789/0 conn(0x7f7fc00a98f0 legacy=0x7f7fc00b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 shutdown_connections 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 >> 172.21.15.44:0/1170275089 conn(0x7f7fc00b7ea0 msgr2=0x7f7fc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 shutdown_connections 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 1 -- 172.21.15.44:0/1170275089 wait complete. 2024-03-25T17:46:17.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:17.215+0000 7f7fc497a640 1 librados: shutdown 2024-03-25T17:46:17.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:46:18.511 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:35.867 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:18.558 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:35.914 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:18.602 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:35.957 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:18.623 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:35.978 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:18.623 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:35.943 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:18.625 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:35.945 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:18.637 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:35.993 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:18.655 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:35.974 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:18.665 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:36.020 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:24.033 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:46:24.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.149+0000 7f0102a25640 10 monclient: get_monmap_and_config 2024-03-25T17:46:24.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.149+0000 7f0102a25640 10 monclient: build_initial_monmap 2024-03-25T17:46:24.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.149+0000 7f0102a25640 10 monclient: monmap: 2024-03-25T17:46:24.150 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:24.150 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:24.150 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:46:24.150848+0000 2024-03-25T17:46:24.151 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:46:24.150848+0000 2024-03-25T17:46:24.151 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:24.151 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:24.151 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:24.151 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:46:24.151 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:46:24.151 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.150+0000 7f0102a25640 1 Processor -- start 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.150+0000 7f0102a25640 1 -- start start 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.150+0000 7f0102a25640 10 monclient: init 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 10 monclient(hunting): picked mon.noname-a con 0x7f00fc0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 10 monclient(hunting): picked mon.noname-b con 0x7f00fc0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 10 monclient(hunting): picked mon.noname-c con 0x7f00fc14be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc0af940 con 0x7f00fc0b4d60 2024-03-25T17:46:24.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc0b0a70 con 0x7f00fc14be90 2024-03-25T17:46:24.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc0b0d50 con 0x7f00fc0b1730 2024-03-25T17:46:24.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:24.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.151+0000 7f0102a25640 10 monclient(hunting): authenticate will time out at 6387.379275s 2024-03-25T17:46:24.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.152+0000 7f00fb7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f00fc0b1730 0x7f00fc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:51840/0 (socket says 172.21.15.44:51840) 2024-03-25T17:46:24.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.152+0000 7f00fb7fe640 1 -- 172.21.15.44:0/3185725525 learned_addr learned my addr 172.21.15.44:0/3185725525 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:24.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.232+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697425947 0 0) 0x7f00fc0af940 con 0x7f00fc0b4d60 2024-03-25T17:46:24.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.232+0000 7f00faffd640 10 monclient(hunting): _negotiate 2024-03-25T17:46:24.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.232+0000 7f00faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:24.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.232+0000 7f00faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:24.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.232+0000 7f00faffd640 10 monclient(hunting): my global_id is 4557 2024-03-25T17:46:24.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.232+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e4003520 con 0x7f00fc0b4d60 2024-03-25T17:46:24.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899450204 0 0) 0x7f00e4003520 con 0x7f00fc0b4d60 2024-03-25T17:46:24.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 >> v1:172.21.15.44:6790/0 conn(0x7f00fc0b1730 legacy=0x7f00fc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 >> v1:172.21.15.111:6789/0 conn(0x7f00fc14be90 legacy=0x7f00fc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 10 monclient: _finish_hunting 0 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 1 monclient: found mon.noname-a 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc0b1030 con 0x7f00fc0b4d60 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 10 monclient: _finish_auth 0 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 10 monclient: _check_auth_tickets 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f00ec001690 con 0x7f00fc0b4d60 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f0102a25640 5 monclient: authenticate success, global_id 4557 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f0102a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f00faffd640 10 monclient: dump: 2024-03-25T17:46:24.251 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:24.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.250+0000 7f0102a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:24.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00ec0017f0 con 0x7f00fc0b4d60 2024-03-25T17:46:24.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f00faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f00faffd640 1 -- 172.21.15.44:0/3185725525 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f00ec001c00 con 0x7f00fc0b4d60 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f00faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f00faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f00faffd640 10 monclient: dump: 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f0102a25640 10 monclient: get_monmap_and_config success 2024-03-25T17:46:24.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f0102a25640 4 set_mon_vals no callback set 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f0102a25640 10 monclient: shutdown 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f0102a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.259+0000 7f0102a25640 1 -- 172.21.15.44:0/3185725525 >> v1:172.21.15.44:6789/0 conn(0x7f00fc0b4d60 legacy=0x7f00fc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.260+0000 7f0102a25640 1 -- 172.21.15.44:0/3185725525 shutdown_connections 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.260+0000 7f0102a25640 1 -- 172.21.15.44:0/3185725525 >> 172.21.15.44:0/3185725525 conn(0x7f00fc0617a0 msgr2=0x7f00fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.260+0000 7f0102a25640 1 -- 172.21.15.44:0/3185725525 shutdown_connections 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.260+0000 7f0102a25640 1 -- 172.21.15.44:0/3185725525 wait complete. 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 10 monclient: build_initial_monmap 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 10 monclient: monmap: 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:46:24.262 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 1 librados: starting msgr at 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 1 librados: starting objecter 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 1 Processor -- start 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 1 -- start start 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 1 librados: setting wanted keys 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 1 librados: calling monclient init 2024-03-25T17:46:24.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.261+0000 7f0102a25640 10 monclient: init 2024-03-25T17:46:24.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:24.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:46:24.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 10 monclient(hunting): picked mon.noname-a con 0x7f00fc0b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:24.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 10 monclient(hunting): picked mon.noname-c con 0x7f00fc0b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:24.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 10 monclient(hunting): picked mon.noname-b con 0x7f00fc14be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:24.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc13a940 con 0x7f00fc0b1730 2024-03-25T17:46:24.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc13ab90 con 0x7f00fc14be90 2024-03-25T17:46:24.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc1d7840 con 0x7f00fc0b4d60 2024-03-25T17:46:24.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:24.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.262+0000 7f0102a25640 10 monclient(hunting): authenticate will time out at 6387.490551s 2024-03-25T17:46:24.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.280+0000 7f00fbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f00fc0b1730 0x7f00fc13bb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51730/0 (socket says 172.21.15.44:51730) 2024-03-25T17:46:24.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.280+0000 7f00fbfff640 1 -- 172.21.15.44:0/448683322 learned_addr learned my addr 172.21.15.44:0/448683322 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:24.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.286+0000 7f00f8ff9640 10 client.?.objecter ms_handle_connect 0x7f00fc0b1730 2024-03-25T17:46:24.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.286+0000 7f00f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:24.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.297+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897121348 0 0) 0x7f00fc13a940 con 0x7f00fc0b1730 2024-03-25T17:46:24.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.297+0000 7f00f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:46:24.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.297+0000 7f00f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:24.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.297+0000 7f00f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:24.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.297+0000 7f00f8ff9640 10 monclient(hunting): my global_id is 4560 2024-03-25T17:46:24.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.297+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e8004eb0 con 0x7f00fc0b1730 2024-03-25T17:46:24.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.308+0000 7f00f8ff9640 10 client.?.objecter ms_handle_connect 0x7f00fc14be90 2024-03-25T17:46:24.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.308+0000 7f00f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:24.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.320+0000 7f00f8ff9640 10 client.?.objecter ms_handle_connect 0x7f00fc0b4d60 2024-03-25T17:46:24.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.320+0000 7f00f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:24.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.351+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1542349467 0 0) 0x7f00e8004eb0 con 0x7f00fc0b1730 2024-03-25T17:46:24.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.351+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00fc13a940 con 0x7f00fc0b1730 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.351+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f00ec001690 con 0x7f00fc0b1730 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.351+0000 7f00f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.351+0000 7f00f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.351+0000 7f00f8ff9640 10 monclient(hunting): dump: 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:24.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.351+0000 7f00f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:46:24.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.374+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1322572364 0 0) 0x7f00fc13a940 con 0x7f00fc0b1730 2024-03-25T17:46:24.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.374+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 >> v1:172.21.15.44:6790/0 conn(0x7f00fc0b4d60 legacy=0x7f00fc1d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:24.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 >> v1:172.21.15.111:6789/0 conn(0x7f00fc14be90 legacy=0x7f00fc1e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:24.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 1 monclient: found mon.a 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc13c240 con 0x7f00fc0b1730 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 10 monclient: _finish_auth 0 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f00f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 5 monclient: authenticate success, global_id 4560 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 10 monclient: _renew_subs 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc139ce0 con 0x7f00fc0b1730 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 10 monclient: _renew_subs 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc1d8650 con 0x7f00fc0b1730 2024-03-25T17:46:24.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.375+0000 7f0102a25640 1 librados: init done 2024-03-25T17:46:24.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.376+0000 7f0102a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:24.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.376+0000 7f0102a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:24.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.376+0000 7f0102a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:24.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.376+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc1401f0 con 0x7f00fc0b1730 2024-03-25T17:46:24.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.774+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00ec0038a0 con 0x7f00fc0b1730 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.774+0000 7f00f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.774+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f00ec003cf0 con 0x7f00fc0b1730 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.774+0000 7f00f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.774+0000 7f00f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.774+0000 7f00f8ff9640 10 monclient: dump: 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:24.775 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:24.776 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:24.776 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:24.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.774+0000 7f00fa7fc640 4 set_mon_vals no callback set 2024-03-25T17:46:24.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.775+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f00ec003f10 con 0x7f00fc0b1730 2024-03-25T17:46:24.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.775+0000 7f00fb7fe640 10 monclient: get_auth_request con 0x7f00e806dc70 auth_method 0 2024-03-25T17:46:24.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.776+0000 7f00f8ff9640 10 client.4560.objecter ms_handle_connect 0x7f00e806dc70 2024-03-25T17:46:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.782+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f00ec0549f0 con 0x7f00fc0b1730 2024-03-25T17:46:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.782+0000 7f00f8ff9640 10 client.4560.objecter ms_dispatch 0x7f00fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:46:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.782+0000 7f00f8ff9640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:46:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.782+0000 7f00f8ff9640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:46:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.782+0000 7f00f8ff9640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-25T17:46:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.782+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00ec050500 con 0x7f00fc0b1730 2024-03-25T17:46:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.782+0000 7f00f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:24.782+0000 7f00f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:46:25.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.049+0000 7f0102a25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:25.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.049+0000 7f0102a25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:25.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.049+0000 7f0102a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:46:25.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.049+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00fc0627b0 con 0x7f00fc0b1730 2024-03-25T17:46:25.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.100+0000 7f00f8ff9640 1 -- 172.21.15.44:0/448683322 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f00ec052430 con 0x7f00fc0b1730 2024-03-25T17:46:25.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.100+0000 7f00f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:25.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.100+0000 7f00f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:46:25.101 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:46:25.101 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 10 librados: watch_flush enter 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 10 librados: watch_flush exit 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 10 client.4560.objecter successfully canceled tick 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 >> v1:172.21.15.44:6800/3257043132 conn(0x7f00e806dc70 legacy=0x7f00e8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 10 monclient: shutdown 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 >> v1:172.21.15.44:6789/0 conn(0x7f00fc0b1730 legacy=0x7f00fc13bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 shutdown_connections 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 >> 172.21.15.44:0/448683322 conn(0x7f00fc0617a0 msgr2=0x7f00fc137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 shutdown_connections 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 1 -- 172.21.15.44:0/448683322 wait complete. 2024-03-25T17:46:25.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:25.103+0000 7f0102a25640 1 librados: shutdown 2024-03-25T17:46:25.115 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:46:31.921 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.039+0000 7f9f65c0f640 10 monclient: get_monmap_and_config 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.039+0000 7f9f65c0f640 10 monclient: build_initial_monmap 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.039+0000 7f9f65c0f640 10 monclient: monmap: 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:46:32.040435+0000 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:46:32.040435+0000 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:32.040 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:46:32.041 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:46:32.041 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:32.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.040+0000 7f9f65c0f640 1 Processor -- start 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.040+0000 7f9f65c0f640 1 -- start start 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.040+0000 7f9f65c0f640 10 monclient: init 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f9f600a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f9f6013f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f9f601430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f60142520 con 0x7f9f600a98f0 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600b28a0 con 0x7f9f6013f710 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600b2a30 con 0x7f9f601430c0 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:32.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.041+0000 7f9f65c0f640 10 monclient(hunting): authenticate will time out at 6395.269140s 2024-03-25T17:46:32.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.047+0000 7f9f5f7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9f600a98f0 0x7f9f600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33206/0 (socket says 172.21.15.44:33206) 2024-03-25T17:46:32.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.047+0000 7f9f5f7fe640 1 -- 172.21.15.44:0/3828779066 learned_addr learned my addr 172.21.15.44:0/3828779066 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:32.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.061+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750750791 0 0) 0x7f9f600b28a0 con 0x7f9f6013f710 2024-03-25T17:46:32.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.061+0000 7f9f5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:46:32.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.061+0000 7f9f5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:32.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.061+0000 7f9f5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:32.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.061+0000 7f9f5e7fc640 10 monclient(hunting): my global_id is 4543 2024-03-25T17:46:32.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.061+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f48003520 con 0x7f9f6013f710 2024-03-25T17:46:32.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.063+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756524195 0 0) 0x7f9f60142520 con 0x7f9f600a98f0 2024-03-25T17:46:32.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.063+0000 7f9f5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:46:32.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.063+0000 7f9f5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:32.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.063+0000 7f9f5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:32.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.063+0000 7f9f5e7fc640 10 monclient(hunting): my global_id is 4563 2024-03-25T17:46:32.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.063+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f600b28a0 con 0x7f9f600a98f0 2024-03-25T17:46:32.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.072+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413720555 0 0) 0x7f9f600b2a30 con 0x7f9f601430c0 2024-03-25T17:46:32.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.072+0000 7f9f5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:46:32.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.072+0000 7f9f5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:32.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.072+0000 7f9f5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:32.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.072+0000 7f9f5e7fc640 10 monclient(hunting): my global_id is 4553 2024-03-25T17:46:32.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.072+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f60142520 con 0x7f9f601430c0 2024-03-25T17:46:32.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539123517 0 0) 0x7f9f48003520 con 0x7f9f6013f710 2024-03-25T17:46:32.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 >> v1:172.21.15.44:6790/0 conn(0x7f9f601430c0 legacy=0x7f9f6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 >> v1:172.21.15.44:6789/0 conn(0x7f9f600a98f0 legacy=0x7f9f600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 1 monclient: found mon.noname-c 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600b44c0 con 0x7f9f6013f710 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9f54001690 con 0x7f9f6013f710 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f65c0f640 5 monclient: authenticate success, global_id 4543 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f65c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:46:32.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f5e7fc640 10 monclient: dump: 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:32.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.076+0000 7f9f65c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:32.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.082+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f540017f0 con 0x7f9f6013f710 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.082+0000 7f9f5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.082+0000 7f9f5e7fc640 1 -- 172.21.15.44:0/3828779066 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9f54001c00 con 0x7f9f6013f710 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.082+0000 7f9f5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.082+0000 7f9f5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.082+0000 7f9f5e7fc640 10 monclient: dump: 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:32.083 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:32.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.082+0000 7f9f65c0f640 10 monclient: get_monmap_and_config success 2024-03-25T17:46:32.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.082+0000 7f9f65c0f640 4 set_mon_vals no callback set 2024-03-25T17:46:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.083+0000 7f9f65c0f640 10 monclient: shutdown 2024-03-25T17:46:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.083+0000 7f9f65c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.083+0000 7f9f65c0f640 1 -- 172.21.15.44:0/3828779066 >> v1:172.21.15.111:6789/0 conn(0x7f9f6013f710 legacy=0x7f9f601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.083+0000 7f9f65c0f640 1 -- 172.21.15.44:0/3828779066 shutdown_connections 2024-03-25T17:46:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.083+0000 7f9f65c0f640 1 -- 172.21.15.44:0/3828779066 >> 172.21.15.44:0/3828779066 conn(0x7f9f600b7ea0 msgr2=0x7f9f600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.083+0000 7f9f65c0f640 1 -- 172.21.15.44:0/3828779066 shutdown_connections 2024-03-25T17:46:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.083+0000 7f9f65c0f640 1 -- 172.21.15.44:0/3828779066 wait complete. 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 10 monclient: build_initial_monmap 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 10 monclient: monmap: 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 1 librados: starting msgr at 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 1 librados: starting objecter 2024-03-25T17:46:32.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 1 Processor -- start 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 1 -- start start 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 1 librados: setting wanted keys 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 1 librados: calling monclient init 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.084+0000 7f9f65c0f640 10 monclient: init 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f9f600a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f9f6013f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f9f601430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f601354c0 con 0x7f9f600a98f0 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f601319b0 con 0x7f9f601430c0 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f60135070 con 0x7f9f6013f710 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:32.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.085+0000 7f9f65c0f640 10 monclient(hunting): authenticate will time out at 6395.313324s 2024-03-25T17:46:32.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.090+0000 7f9f5ffff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9f601430c0 0x7f9f601312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:43766/0 (socket says 172.21.15.44:43766) 2024-03-25T17:46:32.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.090+0000 7f9f5ffff640 1 -- 172.21.15.44:0/525911140 learned_addr learned my addr 172.21.15.44:0/525911140 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:32.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.092+0000 7f9f47fff640 10 client.?.objecter ms_handle_connect 0x7f9f601430c0 2024-03-25T17:46:32.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.092+0000 7f9f47fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:32.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.096+0000 7f9f47fff640 10 client.?.objecter ms_handle_connect 0x7f9f600a98f0 2024-03-25T17:46:32.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.096+0000 7f9f47fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:32.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.098+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852891887 0 0) 0x7f9f601319b0 con 0x7f9f601430c0 2024-03-25T17:46:32.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.098+0000 7f9f47fff640 10 monclient(hunting): _negotiate 2024-03-25T17:46:32.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.098+0000 7f9f47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:32.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.098+0000 7f9f47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:32.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.098+0000 7f9f47fff640 10 monclient(hunting): my global_id is 4546 2024-03-25T17:46:32.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.098+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f4c004eb0 con 0x7f9f601430c0 2024-03-25T17:46:32.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.102+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184727353 0 0) 0x7f9f601354c0 con 0x7f9f600a98f0 2024-03-25T17:46:32.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.102+0000 7f9f47fff640 10 monclient(hunting): _negotiate 2024-03-25T17:46:32.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.102+0000 7f9f47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:32.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.102+0000 7f9f47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:32.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.102+0000 7f9f47fff640 10 monclient(hunting): my global_id is 4566 2024-03-25T17:46:32.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.102+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f601319b0 con 0x7f9f600a98f0 2024-03-25T17:46:32.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.103+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068759719 0 0) 0x7f9f4c004eb0 con 0x7f9f601430c0 2024-03-25T17:46:32.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.103+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f601354c0 con 0x7f9f601430c0 2024-03-25T17:46:32.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.103+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9f58001c70 con 0x7f9f601430c0 2024-03-25T17:46:32.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.103+0000 7f9f47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:32.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.103+0000 7f9f47fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:46:32.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.103+0000 7f9f47fff640 10 monclient(hunting): dump: 2024-03-25T17:46:32.104 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:32.104 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:32.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.103+0000 7f9f47fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 181312537 0 0) 0x7f9f601354c0 con 0x7f9f601430c0 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 >> v1:172.21.15.44:6790/0 conn(0x7f9f6013f710 legacy=0x7f9f60130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 >> v1:172.21.15.44:6789/0 conn(0x7f9f600a98f0 legacy=0x7f9f601d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 10 monclient: _finish_hunting 0 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 1 monclient: found mon.b 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f60135200 con 0x7f9f601430c0 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 10 monclient: _finish_auth 0 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 10 monclient: _check_auth_tickets 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:32.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 5 monclient: authenticate success, global_id 4546 2024-03-25T17:46:32.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 10 monclient: _renew_subs 2024-03-25T17:46:32.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:32.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f600b7010 con 0x7f9f601430c0 2024-03-25T17:46:32.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:46:32.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 10 monclient: _renew_subs 2024-03-25T17:46:32.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:32.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6013b890 con 0x7f9f601430c0 2024-03-25T17:46:32.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.111+0000 7f9f65c0f640 1 librados: init done 2024-03-25T17:46:32.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.112+0000 7f9f65c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:32.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.112+0000 7f9f65c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:32.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.112+0000 7f9f65c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:32.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.112+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f60137160 con 0x7f9f601430c0 2024-03-25T17:46:32.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.377+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f58003990 con 0x7f9f601430c0 2024-03-25T17:46:32.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.377+0000 7f9f47fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.377+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9f58003da0 con 0x7f9f601430c0 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.377+0000 7f9f47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.377+0000 7f9f47fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.377+0000 7f9f47fff640 10 monclient: dump: 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:32.379 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:32.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.377+0000 7f9f5dffb640 4 set_mon_vals no callback set 2024-03-25T17:46:32.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.378+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f9f58003fc0 con 0x7f9f601430c0 2024-03-25T17:46:32.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.379+0000 7f9f5f7fe640 10 monclient: get_auth_request con 0x7f9f4c06dd40 auth_method 0 2024-03-25T17:46:32.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.379+0000 7f9f47fff640 10 client.4546.objecter ms_handle_connect 0x7f9f4c06dd40 2024-03-25T17:46:32.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.383+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f9f58054ae0 con 0x7f9f601430c0 2024-03-25T17:46:32.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.383+0000 7f9f47fff640 10 client.4546.objecter ms_dispatch 0x7f9f60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:46:32.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.383+0000 7f9f47fff640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:46:32.384 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.383+0000 7f9f47fff640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:46:32.384 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.383+0000 7f9f47fff640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T17:46:32.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.386+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f580505f0 con 0x7f9f601430c0 2024-03-25T17:46:32.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.386+0000 7f9f47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:32.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.386+0000 7f9f47fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:46:32.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.612+0000 7f9f65c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:32.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.612+0000 7f9f65c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:32.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.612+0000 7f9f65c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:32.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.612+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f600627b0 con 0x7f9f601430c0 2024-03-25T17:46:32.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.634+0000 7f9f47fff640 1 -- 172.21.15.44:0/525911140 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f9f58052520 con 0x7f9f601430c0 2024-03-25T17:46:32.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.634+0000 7f9f47fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:32.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.634+0000 7f9f47fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:46:32.635 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:46:32.635 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:46:32.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 10 librados: watch_flush enter 2024-03-25T17:46:32.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 10 librados: watch_flush exit 2024-03-25T17:46:32.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T17:46:32.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 10 client.4546.objecter successfully canceled tick 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 >> v1:172.21.15.44:6800/3257043132 conn(0x7f9f4c06dd40 legacy=0x7f9f4c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 10 monclient: shutdown 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 >> v1:172.21.15.111:6789/0 conn(0x7f9f601430c0 legacy=0x7f9f601312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 shutdown_connections 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.636+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 >> 172.21.15.44:0/525911140 conn(0x7f9f600b7ea0 msgr2=0x7f9f6013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.637+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 shutdown_connections 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.637+0000 7f9f65c0f640 1 -- 172.21.15.44:0/525911140 wait complete. 2024-03-25T17:46:32.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:32.637+0000 7f9f65c0f640 1 librados: shutdown 2024-03-25T17:46:32.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:46:38.535 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:55.891 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:38.578 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:55.898 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:38.581 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:55.901 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:38.602 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:55.922 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:38.605 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:55.961 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:38.639 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:55.995 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:38.647 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:56.003 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:38.651 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:56.007 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:38.679 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:56.034 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:39.455 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:46:39.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.569+0000 7f6f8f258640 10 monclient: get_monmap_and_config 2024-03-25T17:46:39.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.569+0000 7f6f8f258640 10 monclient: build_initial_monmap 2024-03-25T17:46:39.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.569+0000 7f6f8f258640 10 monclient: monmap: 2024-03-25T17:46:39.570 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:46:39.570839+0000 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:46:39.570839+0000 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:46:39.571 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:39.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.570+0000 7f6f8f258640 1 Processor -- start 2024-03-25T17:46:39.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.570+0000 7f6f8f258640 1 -- start start 2024-03-25T17:46:39.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.570+0000 7f6f8f258640 10 monclient: init 2024-03-25T17:46:39.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:39.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:46:39.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 10 monclient(hunting): picked mon.noname-c con 0x7f6f880a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:39.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 10 monclient(hunting): picked mon.noname-a con 0x7f6f8813f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:39.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 10 monclient(hunting): picked mon.noname-b con 0x7f6f881430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:39.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f88142520 con 0x7f6f8813f710 2024-03-25T17:46:39.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f880b28a0 con 0x7f6f880a98f0 2024-03-25T17:46:39.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f880b2a30 con 0x7f6f881430c0 2024-03-25T17:46:39.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:39.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.571+0000 7f6f8f258640 10 monclient(hunting): authenticate will time out at 6402.799388s 2024-03-25T17:46:39.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.577+0000 7f6f8cff4640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6f880a98f0 0x7f6f880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:33426/0 (socket says 172.21.15.44:33426) 2024-03-25T17:46:39.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.577+0000 7f6f8cff4640 1 -- 172.21.15.44:0/3575120713 learned_addr learned my addr 172.21.15.44:0/3575120713 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:39.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.598+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533494778 0 0) 0x7f6f880b28a0 con 0x7f6f880a98f0 2024-03-25T17:46:39.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.598+0000 7f6f877fe640 10 monclient(hunting): _negotiate 2024-03-25T17:46:39.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.598+0000 7f6f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:39.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.598+0000 7f6f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:39.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.598+0000 7f6f877fe640 10 monclient(hunting): my global_id is 4549 2024-03-25T17:46:39.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.598+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f70003520 con 0x7f6f880a98f0 2024-03-25T17:46:39.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.608+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065153149 0 0) 0x7f6f88142520 con 0x7f6f8813f710 2024-03-25T17:46:39.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.608+0000 7f6f877fe640 10 monclient(hunting): _negotiate 2024-03-25T17:46:39.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.608+0000 7f6f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:39.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.608+0000 7f6f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:39.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.608+0000 7f6f877fe640 10 monclient(hunting): my global_id is 4569 2024-03-25T17:46:39.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.608+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f880b28a0 con 0x7f6f8813f710 2024-03-25T17:46:39.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.617+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145302685 0 0) 0x7f6f880b2a30 con 0x7f6f881430c0 2024-03-25T17:46:39.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.617+0000 7f6f877fe640 10 monclient(hunting): _negotiate 2024-03-25T17:46:39.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.617+0000 7f6f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:39.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.617+0000 7f6f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:39.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.617+0000 7f6f877fe640 10 monclient(hunting): my global_id is 4556 2024-03-25T17:46:39.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.617+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f88142520 con 0x7f6f881430c0 2024-03-25T17:46:39.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3315867631 0 0) 0x7f6f70003520 con 0x7f6f880a98f0 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 >> v1:172.21.15.44:6790/0 conn(0x7f6f881430c0 legacy=0x7f6f8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 >> v1:172.21.15.44:6789/0 conn(0x7f6f8813f710 legacy=0x7f6f881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 10 monclient: _finish_hunting 0 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 1 monclient: found mon.noname-c 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f880b44c0 con 0x7f6f880a98f0 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 10 monclient: _finish_auth 0 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 10 monclient: _check_auth_tickets 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6f78001690 con 0x7f6f880a98f0 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:46:39.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f877fe640 10 monclient: dump: 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f8f258640 5 monclient: authenticate success, global_id 4549 2024-03-25T17:46:39.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.628+0000 7f6f8f258640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:39.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.647+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f780017f0 con 0x7f6f880a98f0 2024-03-25T17:46:39.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.647+0000 7f6f877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:39.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.647+0000 7f6f877fe640 1 -- 172.21.15.44:0/3575120713 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6f78001c00 con 0x7f6f880a98f0 2024-03-25T17:46:39.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.647+0000 7f6f877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:39.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.647+0000 7f6f877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:46:39.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.647+0000 7f6f877fe640 10 monclient: dump: 2024-03-25T17:46:39.648 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.647+0000 7f6f8f258640 10 monclient: get_monmap_and_config success 2024-03-25T17:46:39.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.647+0000 7f6f8f258640 4 set_mon_vals no callback set 2024-03-25T17:46:39.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.648+0000 7f6f8f258640 10 monclient: shutdown 2024-03-25T17:46:39.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.648+0000 7f6f8f258640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:39.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.648+0000 7f6f8f258640 1 -- 172.21.15.44:0/3575120713 >> v1:172.21.15.111:6789/0 conn(0x7f6f880a98f0 legacy=0x7f6f880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:39.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.648+0000 7f6f8f258640 1 -- 172.21.15.44:0/3575120713 shutdown_connections 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.648+0000 7f6f8f258640 1 -- 172.21.15.44:0/3575120713 >> 172.21.15.44:0/3575120713 conn(0x7f6f880b7ea0 msgr2=0x7f6f880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.648+0000 7f6f8f258640 1 -- 172.21.15.44:0/3575120713 shutdown_connections 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.648+0000 7f6f8f258640 1 -- 172.21.15.44:0/3575120713 wait complete. 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.649+0000 7f6f8f258640 10 monclient: build_initial_monmap 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.649+0000 7f6f8f258640 10 monclient: monmap: 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:39.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.649+0000 7f6f8f258640 1 librados: starting msgr at 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.649+0000 7f6f8f258640 1 librados: starting objecter 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.650+0000 7f6f8f258640 1 Processor -- start 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.650+0000 7f6f8f258640 1 -- start start 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.650+0000 7f6f8f258640 1 librados: setting wanted keys 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.650+0000 7f6f8f258640 1 librados: calling monclient init 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.650+0000 7f6f8f258640 10 monclient: init 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 10 monclient(hunting): picked mon.noname-a con 0x7f6f880a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 10 monclient(hunting): picked mon.noname-b con 0x7f6f8813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 10 monclient(hunting): picked mon.noname-c con 0x7f6f881430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f88135590 con 0x7f6f880a98f0 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f88136e50 con 0x7f6f8813f710 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f88135070 con 0x7f6f881430c0 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:39.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.651+0000 7f6f8f258640 10 monclient(hunting): authenticate will time out at 6402.879039s 2024-03-25T17:46:39.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.655+0000 7f6f8d7f5640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f6f881430c0 0x7f6f881e3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:53642/0 (socket says 172.21.15.44:53642) 2024-03-25T17:46:39.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.655+0000 7f6f8d7f5640 1 -- 172.21.15.44:0/4173129093 learned_addr learned my addr 172.21.15.44:0/4173129093 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:39.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.656+0000 7f6f857fa640 10 client.?.objecter ms_handle_connect 0x7f6f881430c0 2024-03-25T17:46:39.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.656+0000 7f6f857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:39.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.659+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421754772 0 0) 0x7f6f88135070 con 0x7f6f881430c0 2024-03-25T17:46:39.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.659+0000 7f6f857fa640 10 monclient(hunting): _negotiate 2024-03-25T17:46:39.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.659+0000 7f6f857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:39.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.659+0000 7f6f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:39.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.659+0000 7f6f857fa640 10 monclient(hunting): my global_id is 4559 2024-03-25T17:46:39.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.659+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f74004eb0 con 0x7f6f881430c0 2024-03-25T17:46:39.673 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.673+0000 7f6f857fa640 10 client.?.objecter ms_handle_connect 0x7f6f880a98f0 2024-03-25T17:46:39.673 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.673+0000 7f6f857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:39.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.674+0000 7f6f857fa640 10 client.?.objecter ms_handle_connect 0x7f6f8813f710 2024-03-25T17:46:39.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.674+0000 7f6f857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:39.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.680+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481577393 0 0) 0x7f6f74004eb0 con 0x7f6f881430c0 2024-03-25T17:46:39.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.680+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f88135070 con 0x7f6f881430c0 2024-03-25T17:46:39.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.680+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6f80001c70 con 0x7f6f881430c0 2024-03-25T17:46:39.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.680+0000 7f6f857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:39.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.680+0000 7f6f857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:46:39.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.680+0000 7f6f857fa640 10 monclient(hunting): dump: 2024-03-25T17:46:39.680 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:39.680 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:39.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.680+0000 7f6f857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:46:39.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.686+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337811648 0 0) 0x7f6f88136e50 con 0x7f6f8813f710 2024-03-25T17:46:39.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.686+0000 7f6f857fa640 10 monclient(hunting): _negotiate 2024-03-25T17:46:39.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.686+0000 7f6f857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:39.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.686+0000 7f6f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:39.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.686+0000 7f6f857fa640 10 monclient(hunting): my global_id is 4552 2024-03-25T17:46:39.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.686+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f74004eb0 con 0x7f6f8813f710 2024-03-25T17:46:39.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2801226262 0 0) 0x7f6f88135070 con 0x7f6f881430c0 2024-03-25T17:46:39.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 >> v1:172.21.15.111:6789/0 conn(0x7f6f8813f710 legacy=0x7f6f881d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:39.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 >> v1:172.21.15.44:6789/0 conn(0x7f6f880a98f0 legacy=0x7f6f88130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:39.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 10 monclient: _finish_hunting 0 2024-03-25T17:46:39.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 1 monclient: found mon.c 2024-03-25T17:46:39.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:39.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f88135d80 con 0x7f6f881430c0 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 10 monclient: _finish_auth 0 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 10 monclient: _check_auth_tickets 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 5 monclient: authenticate success, global_id 4559 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 10 monclient: _renew_subs 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f88131660 con 0x7f6f881430c0 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 10 monclient: _renew_subs 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f880b7a80 con 0x7f6f881430c0 2024-03-25T17:46:39.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.692+0000 7f6f8f258640 1 librados: init done 2024-03-25T17:46:39.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.694+0000 7f6f8f258640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:39.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.694+0000 7f6f8f258640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:39.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.694+0000 7f6f8f258640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:39.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:39.694+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f88131a00 con 0x7f6f881430c0 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.004+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f80003990 con 0x7f6f881430c0 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.004+0000 7f6f857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.004+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f6f80003c40 con 0x7f6f881430c0 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.004+0000 7f6f857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.004+0000 7f6f857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.004+0000 7f6f857fa640 10 monclient: dump: 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:40.005 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:40.006 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:40.006 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:40.006 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:40.006 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:40.006 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:40.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.004+0000 7f6f86ffd640 4 set_mon_vals no callback set 2024-03-25T17:46:40.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.005+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f6f80003e60 con 0x7f6f881430c0 2024-03-25T17:46:40.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.006+0000 7f6f8cff4640 10 monclient: get_auth_request con 0x7f6f7406dd40 auth_method 0 2024-03-25T17:46:40.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.006+0000 7f6f857fa640 10 client.4559.objecter ms_handle_connect 0x7f6f7406dd40 2024-03-25T17:46:40.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.012+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f6f80054a50 con 0x7f6f881430c0 2024-03-25T17:46:40.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.012+0000 7f6f857fa640 10 client.4559.objecter ms_dispatch 0x7f6f88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:46:40.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.012+0000 7f6f857fa640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:46:40.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.012+0000 7f6f857fa640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:46:40.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.012+0000 7f6f857fa640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-25T17:46:40.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.012+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f80050560 con 0x7f6f881430c0 2024-03-25T17:46:40.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.012+0000 7f6f857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:40.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.012+0000 7f6f857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:46:40.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.274+0000 7f6f8f258640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:40.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.274+0000 7f6f8f258640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:40.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.274+0000 7f6f8f258640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:40.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.274+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f88135200 con 0x7f6f881430c0 2024-03-25T17:46:40.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.312+0000 7f6f857fa640 1 -- 172.21.15.44:0/4173129093 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f6f80052490 con 0x7f6f881430c0 2024-03-25T17:46:40.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.312+0000 7f6f857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:40.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.312+0000 7f6f857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:46:40.313 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:46:40.313 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:46:40.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.314+0000 7f6f8f258640 10 librados: watch_flush enter 2024-03-25T17:46:40.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.314+0000 7f6f8f258640 10 librados: watch_flush exit 2024-03-25T17:46:40.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.314+0000 7f6f8f258640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.314+0000 7f6f8f258640 10 client.4559.objecter successfully canceled tick 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 >> v1:172.21.15.44:6800/3257043132 conn(0x7f6f7406dd40 legacy=0x7f6f74070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 10 monclient: shutdown 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 >> v1:172.21.15.44:6790/0 conn(0x7f6f881430c0 legacy=0x7f6f881e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 shutdown_connections 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 >> 172.21.15.44:0/4173129093 conn(0x7f6f880b7ea0 msgr2=0x7f6f88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 shutdown_connections 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 1 -- 172.21.15.44:0/4173129093 wait complete. 2024-03-25T17:46:40.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:40.315+0000 7f6f8f258640 1 librados: shutdown 2024-03-25T17:46:40.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:46:47.136 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:46:47.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.241+0000 7f56af616640 10 monclient: get_monmap_and_config 2024-03-25T17:46:47.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.241+0000 7f56af616640 10 monclient: build_initial_monmap 2024-03-25T17:46:47.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.241+0000 7f56af616640 10 monclient: monmap: 2024-03-25T17:46:47.242 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:47.242 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:47.242 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:46:47.242725+0000 2024-03-25T17:46:47.242 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:46:47.242725+0000 2024-03-25T17:46:47.242 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:47.243 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:47.243 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:47.243 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:46:47.243 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:46:47.243 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:47.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.242+0000 7f56af616640 1 Processor -- start 2024-03-25T17:46:47.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.242+0000 7f56af616640 1 -- start start 2024-03-25T17:46:47.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.242+0000 7f56af616640 10 monclient: init 2024-03-25T17:46:47.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:47.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:46:47.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 10 monclient(hunting): picked mon.noname-c con 0x7f56a80b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:47.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 10 monclient(hunting): picked mon.noname-a con 0x7f56a80b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:47.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 10 monclient(hunting): picked mon.noname-b con 0x7f56a814be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:47.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80af940 con 0x7f56a80b1730 2024-03-25T17:46:47.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80b0a70 con 0x7f56a80b4d60 2024-03-25T17:46:47.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80b0d50 con 0x7f56a814be90 2024-03-25T17:46:47.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:47.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.243+0000 7f56af616640 10 monclient(hunting): authenticate will time out at 6410.470887s 2024-03-25T17:46:47.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.252+0000 7f56ad3b2640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f56a80b4d60 0x7f56a813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:38176/0 (socket says 172.21.15.44:38176) 2024-03-25T17:46:47.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.252+0000 7f56ad3b2640 1 -- 172.21.15.44:0/3951158395 learned_addr learned my addr 172.21.15.44:0/3951158395 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:47.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.275+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101757511 0 0) 0x7f56a80b0a70 con 0x7f56a80b4d60 2024-03-25T17:46:47.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.275+0000 7f569ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:46:47.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.275+0000 7f569ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:47.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.275+0000 7f569ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:47.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.275+0000 7f569ffff640 10 monclient(hunting): my global_id is 4555 2024-03-25T17:46:47.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.275+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5694003520 con 0x7f56a80b4d60 2024-03-25T17:46:47.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.284+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077551407 0 0) 0x7f56a80b0d50 con 0x7f56a814be90 2024-03-25T17:46:47.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.284+0000 7f569ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:46:47.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.284+0000 7f569ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:47.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.284+0000 7f569ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:47.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.284+0000 7f569ffff640 10 monclient(hunting): my global_id is 4562 2024-03-25T17:46:47.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.285+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a80b0a70 con 0x7f56a814be90 2024-03-25T17:46:47.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.289+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4100018181 0 0) 0x7f5694003520 con 0x7f56a80b4d60 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.289+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 >> v1:172.21.15.44:6790/0 conn(0x7f56a814be90 legacy=0x7f56a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 >> v1:172.21.15.44:6789/0 conn(0x7f56a80b1730 legacy=0x7f56a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 1 monclient: found mon.noname-c 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80b1030 con 0x7f56a80b4d60 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 10 monclient: _finish_auth 0 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f56a4001690 con 0x7f56a80b4d60 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:47.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f569ffff640 10 monclient: dump: 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f56af616640 5 monclient: authenticate success, global_id 4555 2024-03-25T17:46:47.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.290+0000 7f56af616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.305+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56a40017f0 con 0x7f56a80b4d60 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.305+0000 7f569ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.305+0000 7f569ffff640 1 -- 172.21.15.44:0/3951158395 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f56a4001c00 con 0x7f56a80b4d60 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.305+0000 7f569ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.305+0000 7f569ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.305+0000 7f569ffff640 10 monclient: dump: 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:47.306 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:47.307 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:47.307 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:47.307 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:47.307 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:47.307 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:47.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.305+0000 7f56af616640 10 monclient: get_monmap_and_config success 2024-03-25T17:46:47.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.305+0000 7f56af616640 4 set_mon_vals no callback set 2024-03-25T17:46:47.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.306+0000 7f56af616640 10 monclient: shutdown 2024-03-25T17:46:47.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.306+0000 7f56af616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:47.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.306+0000 7f56af616640 1 -- 172.21.15.44:0/3951158395 >> v1:172.21.15.111:6789/0 conn(0x7f56a80b4d60 legacy=0x7f56a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.306+0000 7f56af616640 1 -- 172.21.15.44:0/3951158395 shutdown_connections 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.306+0000 7f56af616640 1 -- 172.21.15.44:0/3951158395 >> 172.21.15.44:0/3951158395 conn(0x7f56a80617a0 msgr2=0x7f56a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.306+0000 7f56af616640 1 -- 172.21.15.44:0/3951158395 shutdown_connections 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.306+0000 7f56af616640 1 -- 172.21.15.44:0/3951158395 wait complete. 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.307+0000 7f56af616640 10 monclient: build_initial_monmap 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.307+0000 7f56af616640 10 monclient: monmap: 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:46:47.309 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:46:47.310 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:47.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 1 librados: starting msgr at 2024-03-25T17:46:47.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 1 librados: starting objecter 2024-03-25T17:46:47.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 1 Processor -- start 2024-03-25T17:46:47.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 1 -- start start 2024-03-25T17:46:47.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 1 librados: setting wanted keys 2024-03-25T17:46:47.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 1 librados: calling monclient init 2024-03-25T17:46:47.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 10 monclient: init 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 10 monclient(hunting): picked mon.noname-a con 0x7f56a80b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 10 monclient(hunting): picked mon.noname-c con 0x7f56a80b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 10 monclient(hunting): picked mon.noname-b con 0x7f56a814be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.308+0000 7f56af616640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a8142050 con 0x7f56a80b1730 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.309+0000 7f56af616640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a81dac50 con 0x7f56a814be90 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.309+0000 7f56af616640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a8140d00 con 0x7f56a80b4d60 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.309+0000 7f56af616640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:47.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.309+0000 7f56af616640 10 monclient(hunting): authenticate will time out at 6410.536764s 2024-03-25T17:46:47.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.319+0000 7f56adbb3640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f56a814be90 0x7f56a81e2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:38230/0 (socket says 172.21.15.44:38230) 2024-03-25T17:46:47.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.319+0000 7f56adbb3640 1 -- 172.21.15.44:0/2308897282 learned_addr learned my addr 172.21.15.44:0/2308897282 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:47.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.322+0000 7f569dffb640 10 client.?.objecter ms_handle_connect 0x7f56a814be90 2024-03-25T17:46:47.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.322+0000 7f569dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:47.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.324+0000 7f569dffb640 10 client.?.objecter ms_handle_connect 0x7f56a80b1730 2024-03-25T17:46:47.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.324+0000 7f569dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:47.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.325+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755906125 0 0) 0x7f56a81dac50 con 0x7f56a814be90 2024-03-25T17:46:47.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.325+0000 7f569dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:46:47.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.325+0000 7f569dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:47.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.325+0000 7f569dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:47.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.325+0000 7f569dffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T17:46:47.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.325+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f568c004eb0 con 0x7f56a814be90 2024-03-25T17:46:47.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.326+0000 7f569dffb640 10 client.?.objecter ms_handle_connect 0x7f56a80b4d60 2024-03-25T17:46:47.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.326+0000 7f569dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:47.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.335+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237989276 0 0) 0x7f56a8140d00 con 0x7f56a80b4d60 2024-03-25T17:46:47.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.335+0000 7f569dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:46:47.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.335+0000 7f569dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:47.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.335+0000 7f569dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:47.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.335+0000 7f569dffb640 10 monclient(hunting): my global_id is 4565 2024-03-25T17:46:47.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.335+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a81dac50 con 0x7f56a80b4d60 2024-03-25T17:46:47.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.336+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768425905 0 0) 0x7f568c004eb0 con 0x7f56a814be90 2024-03-25T17:46:47.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.336+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56a8140d00 con 0x7f56a814be90 2024-03-25T17:46:47.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.336+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f56a0001c90 con 0x7f56a814be90 2024-03-25T17:46:47.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.336+0000 7f569dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:47.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.336+0000 7f569dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:46:47.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.336+0000 7f569dffb640 10 monclient(hunting): dump: 2024-03-25T17:46:47.337 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:47.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.336+0000 7f569dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:46:47.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.341+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863975122 0 0) 0x7f56a8142050 con 0x7f56a80b1730 2024-03-25T17:46:47.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.341+0000 7f569dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:46:47.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.341+0000 7f569dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:47.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.341+0000 7f569dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:47.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.341+0000 7f569dffb640 10 monclient(hunting): my global_id is 4578 2024-03-25T17:46:47.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.341+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f568c004eb0 con 0x7f56a80b1730 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3429626538 0 0) 0x7f56a8140d00 con 0x7f56a814be90 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 >> v1:172.21.15.44:6790/0 conn(0x7f56a80b4d60 legacy=0x7f56a81db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 >> v1:172.21.15.44:6789/0 conn(0x7f56a80b1730 legacy=0x7f56a81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 1 monclient: found mon.b 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a81423e0 con 0x7f56a814be90 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 10 monclient: _finish_auth 0 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:46:47.349 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f569dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 5 monclient: authenticate success, global_id 4558 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 10 monclient: _renew_subs 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a813b470 con 0x7f56a814be90 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 10 monclient: _renew_subs 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a81dba40 con 0x7f56a814be90 2024-03-25T17:46:47.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.348+0000 7f56af616640 1 librados: init done 2024-03-25T17:46:47.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.349+0000 7f56af616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:47.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.349+0000 7f56af616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:47.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.349+0000 7f56af616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:47.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.349+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a81401f0 con 0x7f56a814be90 2024-03-25T17:46:47.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.613+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56a0003990 con 0x7f56a814be90 2024-03-25T17:46:47.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.613+0000 7f569dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.613+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f56a0003d60 con 0x7f56a814be90 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.613+0000 7f569dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.613+0000 7f569dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.613+0000 7f569dffb640 10 monclient: dump: 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:47.614 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:47.615 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:47.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.613+0000 7f569f7fe640 4 set_mon_vals no callback set 2024-03-25T17:46:47.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.614+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f56a0003f40 con 0x7f56a814be90 2024-03-25T17:46:47.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.614+0000 7f56ad3b2640 10 monclient: get_auth_request con 0x7f568c06de20 auth_method 0 2024-03-25T17:46:47.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.615+0000 7f569dffb640 10 client.4558.objecter ms_handle_connect 0x7f568c06de20 2024-03-25T17:46:47.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.619+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f56a0054aa0 con 0x7f56a814be90 2024-03-25T17:46:47.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.619+0000 7f569dffb640 10 client.4558.objecter ms_dispatch 0x7f56a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:46:47.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.619+0000 7f569dffb640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:46:47.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.619+0000 7f569dffb640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:46:47.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.619+0000 7f569dffb640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T17:46:47.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.619+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56a00505b0 con 0x7f56a814be90 2024-03-25T17:46:47.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.619+0000 7f569dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:47.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.619+0000 7f569dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:46:47.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.877+0000 7f56af616640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:47.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.877+0000 7f56af616640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:47.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.877+0000 7f56af616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:46:47.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.877+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56a8140e90 con 0x7f56a814be90 2024-03-25T17:46:47.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.903+0000 7f569dffb640 1 -- 172.21.15.44:0/2308897282 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f56a00524e0 con 0x7f56a814be90 2024-03-25T17:46:47.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.903+0000 7f569dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:47.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.903+0000 7f569dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:46:47.904 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:46:47.904 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:46:47.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.905+0000 7f56af616640 10 librados: watch_flush enter 2024-03-25T17:46:47.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.905+0000 7f56af616640 10 librados: watch_flush exit 2024-03-25T17:46:47.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.905+0000 7f56af616640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T17:46:47.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.905+0000 7f56af616640 10 client.4558.objecter successfully canceled tick 2024-03-25T17:46:47.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 >> v1:172.21.15.44:6800/3257043132 conn(0x7f568c06de20 legacy=0x7f568c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:47.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 10 monclient: shutdown 2024-03-25T17:46:47.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:47.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 >> v1:172.21.15.111:6789/0 conn(0x7f56a814be90 legacy=0x7f56a81e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:47.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 shutdown_connections 2024-03-25T17:46:47.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 >> 172.21.15.44:0/2308897282 conn(0x7f56a80617a0 msgr2=0x7f56a80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:47.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 shutdown_connections 2024-03-25T17:46:47.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 1 -- 172.21.15.44:0/2308897282 wait complete. 2024-03-25T17:46:47.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:47.906+0000 7f56af616640 1 librados: shutdown 2024-03-25T17:46:47.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:46:54.725 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:46:54.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.845+0000 7fec66f58640 10 monclient: get_monmap_and_config 2024-03-25T17:46:54.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.845+0000 7fec66f58640 10 monclient: build_initial_monmap 2024-03-25T17:46:54.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.845+0000 7fec66f58640 10 monclient: monmap: 2024-03-25T17:46:54.847 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:54.847 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:54.847 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:46:54.847611+0000 2024-03-25T17:46:54.847 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:46:54.847611+0000 2024-03-25T17:46:54.847 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:54.848 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:54.848 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:54.848 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:46:54.848 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:46:54.848 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:54.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.846+0000 7fec66f58640 1 Processor -- start 2024-03-25T17:46:54.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.846+0000 7fec66f58640 1 -- start start 2024-03-25T17:46:54.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.846+0000 7fec66f58640 10 monclient: init 2024-03-25T17:46:54.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:54.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:46:54.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 10 monclient(hunting): picked mon.noname-a con 0x7fec600a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:54.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 10 monclient(hunting): picked mon.noname-b con 0x7fec6013f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:54.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 10 monclient(hunting): picked mon.noname-c con 0x7fec601430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:54.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec60142520 con 0x7fec600a98f0 2024-03-25T17:46:54.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600b28a0 con 0x7fec601430c0 2024-03-25T17:46:54.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600b2a30 con 0x7fec6013f710 2024-03-25T17:46:54.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:54.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.847+0000 7fec66f58640 10 monclient(hunting): authenticate will time out at 6418.076140s 2024-03-25T17:46:54.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.855+0000 7fec5ffff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fec6013f710 0x7fec601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:52304/0 (socket says 172.21.15.44:52304) 2024-03-25T17:46:54.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.855+0000 7fec5ffff640 1 -- 172.21.15.44:0/3419182962 learned_addr learned my addr 172.21.15.44:0/3419182962 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:54.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.868+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212095548 0 0) 0x7fec600b2a30 con 0x7fec6013f710 2024-03-25T17:46:54.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.869+0000 7fec5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:46:54.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.869+0000 7fec5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:54.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.869+0000 7fec5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:54.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.869+0000 7fec5f7fe640 10 monclient(hunting): my global_id is 4568 2024-03-25T17:46:54.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.869+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec50003520 con 0x7fec6013f710 2024-03-25T17:46:54.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.874+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434894496 0 0) 0x7fec600b28a0 con 0x7fec601430c0 2024-03-25T17:46:54.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.874+0000 7fec5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:46:54.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.874+0000 7fec5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:54.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.874+0000 7fec5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:54.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.874+0000 7fec5f7fe640 10 monclient(hunting): my global_id is 4561 2024-03-25T17:46:54.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.874+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec600b2a30 con 0x7fec601430c0 2024-03-25T17:46:54.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2207337347 0 0) 0x7fec50003520 con 0x7fec6013f710 2024-03-25T17:46:54.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 >> v1:172.21.15.111:6789/0 conn(0x7fec601430c0 legacy=0x7fec6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:54.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 >> v1:172.21.15.44:6789/0 conn(0x7fec600a98f0 legacy=0x7fec600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:46:54.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:46:54.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 1 monclient: found mon.noname-b 2024-03-25T17:46:54.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:54.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:46:54.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec600b44c0 con 0x7fec6013f710 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fec4c001690 con 0x7fec6013f710 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec5f7fe640 10 monclient: dump: 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:54.880 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:54.881 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:54.881 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:54.881 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:54.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec66f58640 5 monclient: authenticate success, global_id 4568 2024-03-25T17:46:54.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.877+0000 7fec66f58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.904+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec4c0017f0 con 0x7fec6013f710 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.904+0000 7fec5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.904+0000 7fec5f7fe640 1 -- 172.21.15.44:0/3419182962 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fec4c001c00 con 0x7fec6013f710 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.904+0000 7fec5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.904+0000 7fec5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.904+0000 7fec5f7fe640 10 monclient: dump: 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:54.906 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:54.907 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:54.907 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:54.907 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:54.907 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:54.907 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:54.907 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:54.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.904+0000 7fec66f58640 10 monclient: get_monmap_and_config success 2024-03-25T17:46:54.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.904+0000 7fec66f58640 4 set_mon_vals no callback set 2024-03-25T17:46:54.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.905+0000 7fec66f58640 10 monclient: shutdown 2024-03-25T17:46:54.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.905+0000 7fec66f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:54.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.905+0000 7fec66f58640 1 -- 172.21.15.44:0/3419182962 >> v1:172.21.15.44:6790/0 conn(0x7fec6013f710 legacy=0x7fec601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:54.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.905+0000 7fec66f58640 1 -- 172.21.15.44:0/3419182962 shutdown_connections 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.905+0000 7fec66f58640 1 -- 172.21.15.44:0/3419182962 >> 172.21.15.44:0/3419182962 conn(0x7fec600b7ea0 msgr2=0x7fec600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.905+0000 7fec66f58640 1 -- 172.21.15.44:0/3419182962 shutdown_connections 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.905+0000 7fec66f58640 1 -- 172.21.15.44:0/3419182962 wait complete. 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.906+0000 7fec66f58640 10 monclient: build_initial_monmap 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.906+0000 7fec66f58640 10 monclient: monmap: 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:46:54.909 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:54.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.906+0000 7fec66f58640 1 librados: starting msgr at 2024-03-25T17:46:54.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.906+0000 7fec66f58640 1 librados: starting objecter 2024-03-25T17:46:54.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 1 Processor -- start 2024-03-25T17:46:54.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 1 -- start start 2024-03-25T17:46:54.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 1 librados: setting wanted keys 2024-03-25T17:46:54.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 1 librados: calling monclient init 2024-03-25T17:46:54.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 10 monclient: init 2024-03-25T17:46:54.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 10 monclient: _reopen_session rank -1 2024-03-25T17:46:54.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:46:54.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 10 monclient(hunting): picked mon.noname-c con 0x7fec600a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:46:54.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 10 monclient(hunting): picked mon.noname-a con 0x7fec6013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:46:54.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 10 monclient(hunting): picked mon.noname-b con 0x7fec601430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:46:54.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec601d7a10 con 0x7fec6013f710 2024-03-25T17:46:54.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec601d76e0 con 0x7fec601430c0 2024-03-25T17:46:54.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.907+0000 7fec66f58640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec601d5210 con 0x7fec600a98f0 2024-03-25T17:46:54.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.908+0000 7fec66f58640 10 monclient(hunting): _renew_subs 2024-03-25T17:46:54.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.908+0000 7fec66f58640 10 monclient(hunting): authenticate will time out at 6418.136692s 2024-03-25T17:46:54.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.910+0000 7fec64cf4640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fec600a98f0 0x7fec601d7f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:52336/0 (socket says 172.21.15.44:52336) 2024-03-25T17:46:54.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.910+0000 7fec64cf4640 1 -- 172.21.15.44:0/2961298534 learned_addr learned my addr 172.21.15.44:0/2961298534 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:46:54.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.912+0000 7fec5d7fa640 10 client.?.objecter ms_handle_connect 0x7fec600a98f0 2024-03-25T17:46:54.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.912+0000 7fec5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:54.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.921+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958178291 0 0) 0x7fec601d5210 con 0x7fec600a98f0 2024-03-25T17:46:54.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.921+0000 7fec5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:46:54.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.921+0000 7fec5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:54.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.921+0000 7fec5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:54.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.921+0000 7fec5d7fa640 10 monclient(hunting): my global_id is 4571 2024-03-25T17:46:54.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.921+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec44004eb0 con 0x7fec600a98f0 2024-03-25T17:46:54.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.931+0000 7fec5d7fa640 10 client.?.objecter ms_handle_connect 0x7fec6013f710 2024-03-25T17:46:54.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.931+0000 7fec5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:54.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.933+0000 7fec5d7fa640 10 client.?.objecter ms_handle_connect 0x7fec601430c0 2024-03-25T17:46:54.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.933+0000 7fec5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.935+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2025941824 0 0) 0x7fec44004eb0 con 0x7fec600a98f0 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.935+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec601d5210 con 0x7fec600a98f0 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.935+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fec54004080 con 0x7fec600a98f0 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.935+0000 7fec5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.935+0000 7fec5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.935+0000 7fec5d7fa640 10 monclient(hunting): dump: 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:54.937 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:54.938 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:54.938 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:54.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.935+0000 7fec5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:46:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.944+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883053772 0 0) 0x7fec601d76e0 con 0x7fec601430c0 2024-03-25T17:46:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.944+0000 7fec5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:46:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.944+0000 7fec5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.944+0000 7fec5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.944+0000 7fec5d7fa640 10 monclient(hunting): my global_id is 4564 2024-03-25T17:46:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.944+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec44004eb0 con 0x7fec601430c0 2024-03-25T17:46:54.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.945+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594682662 0 0) 0x7fec601d7a10 con 0x7fec6013f710 2024-03-25T17:46:54.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.945+0000 7fec5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:46:54.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.945+0000 7fec5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:46:54.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.945+0000 7fec5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:46:54.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.945+0000 7fec5d7fa640 10 monclient(hunting): my global_id is 4581 2024-03-25T17:46:54.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.945+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec601d76e0 con 0x7fec6013f710 2024-03-25T17:46:54.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2708300288 0 0) 0x7fec601d5210 con 0x7fec600a98f0 2024-03-25T17:46:54.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 >> v1:172.21.15.111:6789/0 conn(0x7fec601430c0 legacy=0x7fec601384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:54.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 >> v1:172.21.15.44:6789/0 conn(0x7fec6013f710 legacy=0x7fec601d4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:54.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:46:54.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 1 monclient: found mon.c 2024-03-25T17:46:54.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:46:54.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:54.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec601d41e0 con 0x7fec600a98f0 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 10 monclient: _finish_auth 0 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 5 monclient: authenticate success, global_id 4571 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 10 monclient: _renew_subs 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec601d4450 con 0x7fec600a98f0 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 10 monclient: _renew_subs 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec60138dd0 con 0x7fec600a98f0 2024-03-25T17:46:54.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.949+0000 7fec66f58640 1 librados: init done 2024-03-25T17:46:54.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.950+0000 7fec66f58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:54.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.950+0000 7fec66f58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:54.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.950+0000 7fec66f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:54.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:54.951+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec601d7870 con 0x7fec600a98f0 2024-03-25T17:46:55.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.258+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec54000ed0 con 0x7fec600a98f0 2024-03-25T17:46:55.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.258+0000 7fec5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:46:55.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.258+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fec540010c0 con 0x7fec600a98f0 2024-03-25T17:46:55.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.258+0000 7fec5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.258+0000 7fec5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.258+0000 7fec5d7fa640 10 monclient: dump: 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:46:55.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.258+0000 7fec5effd640 4 set_mon_vals no callback set 2024-03-25T17:46:55.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.259+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fec540012e0 con 0x7fec600a98f0 2024-03-25T17:46:55.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.259+0000 7fec5ffff640 10 monclient: get_auth_request con 0x7fec4406de60 auth_method 0 2024-03-25T17:46:55.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.259+0000 7fec5d7fa640 10 client.4571.objecter ms_handle_connect 0x7fec4406de60 2024-03-25T17:46:55.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.265+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fec54054650 con 0x7fec600a98f0 2024-03-25T17:46:55.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.265+0000 7fec5d7fa640 10 client.4571.objecter ms_dispatch 0x7fec60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:46:55.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.265+0000 7fec5d7fa640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:46:55.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.265+0000 7fec5d7fa640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:46:55.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.265+0000 7fec5d7fa640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-25T17:46:55.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.265+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec54050160 con 0x7fec600a98f0 2024-03-25T17:46:55.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.265+0000 7fec5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:46:55.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.265+0000 7fec5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:46:55.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.489+0000 7fec66f58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:55.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.489+0000 7fec66f58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:55.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.489+0000 7fec66f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:46:55.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.489+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec600627b0 con 0x7fec600a98f0 2024-03-25T17:46:55.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.508+0000 7fec5d7fa640 1 -- 172.21.15.44:0/2961298534 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fec54052090 con 0x7fec600a98f0 2024-03-25T17:46:55.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.508+0000 7fec5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:46:55.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.508+0000 7fec5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:46:55.510 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:46:55.510 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 10 librados: watch_flush enter 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 10 librados: watch_flush exit 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 10 client.4571.objecter successfully canceled tick 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 >> v1:172.21.15.44:6800/3257043132 conn(0x7fec4406de60 legacy=0x7fec44070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 10 monclient: shutdown 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 >> v1:172.21.15.44:6790/0 conn(0x7fec600a98f0 legacy=0x7fec601d7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 shutdown_connections 2024-03-25T17:46:55.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 >> 172.21.15.44:0/2961298534 conn(0x7fec600b7ea0 msgr2=0x7fec60074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:46:55.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 shutdown_connections 2024-03-25T17:46:55.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 1 -- 172.21.15.44:0/2961298534 wait complete. 2024-03-25T17:46:55.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:46:55.511+0000 7fec66f58640 1 librados: shutdown 2024-03-25T17:46:55.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:46:58.640 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:15.996 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:58.655 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:16.011 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:58.691 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:16.046 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:58.706 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:16.061 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:58.710 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:16.030 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:58.714 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:16.070 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:58.715 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:16.035 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:58.743 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:16.063 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:46:58.745 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:16.101 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:02.331 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:47:02.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.430+0000 7f93b68ff640 10 monclient: get_monmap_and_config 2024-03-25T17:47:02.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.430+0000 7f93b68ff640 10 monclient: build_initial_monmap 2024-03-25T17:47:02.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.430+0000 7f93b68ff640 10 monclient: monmap: 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:47:02.431046+0000 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:47:02.431046+0000 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:47:02.431 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 1 Processor -- start 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 1 -- start start 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 10 monclient: init 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 10 monclient(hunting): picked mon.noname-a con 0x7f93b00b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 10 monclient(hunting): picked mon.noname-c con 0x7f93b00b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 10 monclient(hunting): picked mon.noname-b con 0x7f93b014be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b00af940 con 0x7f93b00b4d60 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b00b0a70 con 0x7f93b00b1730 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b00b0d50 con 0x7f93b014be90 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:02.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.431+0000 7f93b68ff640 10 monclient(hunting): authenticate will time out at 6425.659463s 2024-03-25T17:47:02.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.443+0000 7f93affff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f93b00b4d60 0x7f93b013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:59498/0 (socket says 172.21.15.44:59498) 2024-03-25T17:47:02.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.443+0000 7f93affff640 1 -- 172.21.15.44:0/1695279259 learned_addr learned my addr 172.21.15.44:0/1695279259 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:02.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.458+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352325554 0 0) 0x7f93b00af940 con 0x7f93b00b4d60 2024-03-25T17:47:02.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.458+0000 7f93af7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:47:02.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.458+0000 7f93af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:02.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.458+0000 7f93af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:02.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.458+0000 7f93af7fe640 10 monclient(hunting): my global_id is 4584 2024-03-25T17:47:02.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.458+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f939c003520 con 0x7f93b00b4d60 2024-03-25T17:47:02.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 925444599 0 0) 0x7f939c003520 con 0x7f93b00b4d60 2024-03-25T17:47:02.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 >> v1:172.21.15.44:6790/0 conn(0x7f93b014be90 legacy=0x7f93b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:47:02.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 >> v1:172.21.15.111:6789/0 conn(0x7f93b00b1730 legacy=0x7f93b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:47:02.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:47:02.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 1 monclient: found mon.noname-a 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00b1030 con 0x7f93b00b4d60 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 10 monclient: _finish_auth 0 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f93a0001690 con 0x7f93b00b4d60 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.469+0000 7f93af7fe640 10 monclient: dump: 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.471 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:02.472 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:02.472 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:02.472 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:02.472 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:02.472 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:02.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.470+0000 7f93b68ff640 5 monclient: authenticate success, global_id 4584 2024-03-25T17:47:02.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.470+0000 7f93b68ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:02.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.484+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93a0001860 con 0x7f93b00b4d60 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.484+0000 7f93af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.484+0000 7f93af7fe640 1 -- 172.21.15.44:0/1695279259 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f93a0001cb0 con 0x7f93b00b4d60 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.484+0000 7f93b68ff640 10 monclient: get_monmap_and_config success 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.484+0000 7f93b68ff640 4 set_mon_vals no callback set 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.484+0000 7f93af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.484+0000 7f93af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.484+0000 7f93af7fe640 10 monclient: dump: 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:02.485 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:02.486 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:02.486 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:02.486 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:02.486 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.485+0000 7f93b68ff640 10 monclient: shutdown 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.485+0000 7f93b68ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.485+0000 7f93b68ff640 1 -- 172.21.15.44:0/1695279259 >> v1:172.21.15.44:6789/0 conn(0x7f93b00b4d60 legacy=0x7f93b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.485+0000 7f93b68ff640 1 -- 172.21.15.44:0/1695279259 shutdown_connections 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.485+0000 7f93b68ff640 1 -- 172.21.15.44:0/1695279259 >> 172.21.15.44:0/1695279259 conn(0x7f93b00617a0 msgr2=0x7f93b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.485+0000 7f93b68ff640 1 -- 172.21.15.44:0/1695279259 shutdown_connections 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.485+0000 7f93b68ff640 1 -- 172.21.15.44:0/1695279259 wait complete. 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 10 monclient: build_initial_monmap 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 10 monclient: monmap: 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:47:02.488 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 1 librados: starting msgr at 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 1 librados: starting objecter 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 1 Processor -- start 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 1 -- start start 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 1 librados: setting wanted keys 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 1 librados: calling monclient init 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.486+0000 7f93b68ff640 10 monclient: init 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:47:02.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 10 monclient(hunting): picked mon.noname-a con 0x7f93b00b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:02.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 10 monclient(hunting): picked mon.noname-b con 0x7f93b00b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:02.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 10 monclient(hunting): picked mon.noname-c con 0x7f93b014be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:02.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b01d7940 con 0x7f93b00b1730 2024-03-25T17:47:02.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b013a7d0 con 0x7f93b00b4d60 2024-03-25T17:47:02.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b013b7c0 con 0x7f93b014be90 2024-03-25T17:47:02.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:02.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.487+0000 7f93b68ff640 10 monclient(hunting): authenticate will time out at 6425.715100s 2024-03-25T17:47:02.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.494+0000 7f93b4e9c640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f93b014be90 0x7f93b0132d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:55606/0 (socket says 172.21.15.44:55606) 2024-03-25T17:47:02.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.494+0000 7f93b4e9c640 1 -- 172.21.15.44:0/2149283224 learned_addr learned my addr 172.21.15.44:0/2149283224 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:02.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.495+0000 7f93ad7fa640 10 client.?.objecter ms_handle_connect 0x7f93b014be90 2024-03-25T17:47:02.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.495+0000 7f93ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:02.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.504+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534236636 0 0) 0x7f93b013b7c0 con 0x7f93b014be90 2024-03-25T17:47:02.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.504+0000 7f93ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:47:02.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.504+0000 7f93ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:02.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.504+0000 7f93ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:02.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.504+0000 7f93ad7fa640 10 monclient(hunting): my global_id is 4574 2024-03-25T17:47:02.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.504+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9390004eb0 con 0x7f93b014be90 2024-03-25T17:47:02.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.506+0000 7f93ad7fa640 10 client.?.objecter ms_handle_connect 0x7f93b00b1730 2024-03-25T17:47:02.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.506+0000 7f93ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:02.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.513+0000 7f93ad7fa640 10 client.?.objecter ms_handle_connect 0x7f93b00b4d60 2024-03-25T17:47:02.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.513+0000 7f93ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:02.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.522+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044876600 0 0) 0x7f93b013a7d0 con 0x7f93b00b4d60 2024-03-25T17:47:02.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.522+0000 7f93ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:47:02.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.522+0000 7f93ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:02.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.522+0000 7f93ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:02.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.522+0000 7f93ad7fa640 10 monclient(hunting): my global_id is 4567 2024-03-25T17:47:02.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.522+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b013b7c0 con 0x7f93b00b4d60 2024-03-25T17:47:02.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.525+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77186320 0 0) 0x7f93b01d7940 con 0x7f93b00b1730 2024-03-25T17:47:02.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.525+0000 7f93ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:47:02.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.525+0000 7f93ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:02.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.525+0000 7f93ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:02.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.525+0000 7f93ad7fa640 10 monclient(hunting): my global_id is 4587 2024-03-25T17:47:02.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.526+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b013a7d0 con 0x7f93b00b1730 2024-03-25T17:47:02.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.537+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 674564980 0 0) 0x7f93b013b7c0 con 0x7f93b00b4d60 2024-03-25T17:47:02.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.537+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93b01d7940 con 0x7f93b00b4d60 2024-03-25T17:47:02.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.537+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9398001810 con 0x7f93b00b4d60 2024-03-25T17:47:02.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.537+0000 7f93ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:02.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.537+0000 7f93ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.537+0000 7f93ad7fa640 10 monclient(hunting): dump: 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:02.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.538+0000 7f93ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:47:02.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.546+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4064492884 0 0) 0x7f9390004eb0 con 0x7f93b014be90 2024-03-25T17:47:02.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.546+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93b013b7c0 con 0x7f93b014be90 2024-03-25T17:47:02.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.546+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f93a8001ad0 con 0x7f93b014be90 2024-03-25T17:47:02.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.546+0000 7f93ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:02.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.546+0000 7f93ad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:02.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.546+0000 7f93ad7fa640 10 monclient(hunting): dump: 2024-03-25T17:47:02.546 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:02.547 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:02.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1421396090 0 0) 0x7f93b01d7940 con 0x7f93b00b4d60 2024-03-25T17:47:02.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 >> v1:172.21.15.44:6790/0 conn(0x7f93b014be90 legacy=0x7f93b0132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:02.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 >> v1:172.21.15.44:6789/0 conn(0x7f93b00b1730 legacy=0x7f93b013a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:02.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 1 monclient: found mon.b 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b01d8530 con 0x7f93b00b4d60 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 10 monclient: _finish_auth 0 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93b68ff640 5 monclient: authenticate success, global_id 4567 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93b68ff640 10 monclient: _renew_subs 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93b68ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.551+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b0143900 con 0x7f93b00b4d60 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 10 monclient: _renew_subs 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:02.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b0139620 con 0x7f93b00b4d60 2024-03-25T17:47:02.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 1 librados: init done 2024-03-25T17:47:02.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:02.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:02.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:02.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.552+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b01d77a0 con 0x7f93b00b4d60 2024-03-25T17:47:02.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.829+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9398001d00 con 0x7f93b00b4d60 2024-03-25T17:47:02.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.829+0000 7f93ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:02.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.829+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f93980035f0 con 0x7f93b00b4d60 2024-03-25T17:47:02.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.829+0000 7f93ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:02.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.829+0000 7f93ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:02.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.829+0000 7f93ad7fa640 10 monclient: dump: 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.829+0000 7f93aeffd640 4 set_mon_vals no callback set 2024-03-25T17:47:02.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.829+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f93980195c0 con 0x7f93b00b4d60 2024-03-25T17:47:02.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.830+0000 7f93affff640 10 monclient: get_auth_request con 0x7f939006df20 auth_method 0 2024-03-25T17:47:02.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.830+0000 7f93ad7fa640 10 client.4567.objecter ms_handle_connect 0x7f939006df20 2024-03-25T17:47:02.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.834+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f93980533a0 con 0x7f93b00b4d60 2024-03-25T17:47:02.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.834+0000 7f93ad7fa640 10 client.4567.objecter ms_dispatch 0x7f93b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:47:02.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.834+0000 7f93ad7fa640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:47:02.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.834+0000 7f93ad7fa640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:47:02.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.834+0000 7f93ad7fa640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T17:47:02.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.835+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f939804ece0 con 0x7f93b00b4d60 2024-03-25T17:47:02.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.835+0000 7f93ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:02.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:02.835+0000 7f93ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:47:03.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.086+0000 7f93b68ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:03.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.086+0000 7f93b68ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:03.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.086+0000 7f93b68ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:03.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.086+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93b00b22f0 con 0x7f93b00b4d60 2024-03-25T17:47:03.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.111+0000 7f93ad7fa640 1 -- 172.21.15.44:0/2149283224 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f93b00b22f0 con 0x7f93b00b4d60 2024-03-25T17:47:03.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.111+0000 7f93ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:03.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.111+0000 7f93ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:47:03.112 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:47:03.112 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:47:03.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.113+0000 7f93b68ff640 10 librados: watch_flush enter 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.113+0000 7f93b68ff640 10 librados: watch_flush exit 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.113+0000 7f93b68ff640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.113+0000 7f93b68ff640 10 client.4567.objecter successfully canceled tick 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 >> v1:172.21.15.44:6800/3257043132 conn(0x7f939006df20 legacy=0x7f93900703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 10 monclient: shutdown 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 >> v1:172.21.15.111:6789/0 conn(0x7f93b00b4d60 legacy=0x7f93b01d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 shutdown_connections 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 >> 172.21.15.44:0/2149283224 conn(0x7f93b00617a0 msgr2=0x7f93b00b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 shutdown_connections 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 1 -- 172.21.15.44:0/2149283224 wait complete. 2024-03-25T17:47:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:03.114+0000 7f93b68ff640 1 librados: shutdown 2024-03-25T17:47:03.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:47:09.931 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:47:10.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.047+0000 7f8466255640 10 monclient: get_monmap_and_config 2024-03-25T17:47:10.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.047+0000 7f8466255640 10 monclient: build_initial_monmap 2024-03-25T17:47:10.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.047+0000 7f8466255640 10 monclient: monmap: 2024-03-25T17:47:10.047 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:10.047 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:10.047 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:47:10.047983+0000 2024-03-25T17:47:10.047 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:47:10.047983+0000 2024-03-25T17:47:10.048 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:10.048 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:10.048 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:10.048 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:47:10.048 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:47:10.048 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.047+0000 7f8466255640 1 Processor -- start 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.047+0000 7f8466255640 1 -- start start 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.047+0000 7f8466255640 10 monclient: init 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 10 monclient(hunting): picked mon.noname-b con 0x7f84600a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 10 monclient(hunting): picked mon.noname-a con 0x7f846013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 10 monclient(hunting): picked mon.noname-c con 0x7f84601430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8460142520 con 0x7f846013f710 2024-03-25T17:47:10.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84600b28a0 con 0x7f84601430c0 2024-03-25T17:47:10.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84600b2a30 con 0x7f84600a98f0 2024-03-25T17:47:10.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:10.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.048+0000 7f8466255640 10 monclient(hunting): authenticate will time out at 6433.276218s 2024-03-25T17:47:10.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.050+0000 7f845ffff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f84601430c0 0x7f846014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:58822/0 (socket says 172.21.15.44:58822) 2024-03-25T17:47:10.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.050+0000 7f845ffff640 1 -- 172.21.15.44:0/3898207204 learned_addr learned my addr 172.21.15.44:0/3898207204 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:10.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.056+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672440825 0 0) 0x7f84600b28a0 con 0x7f84601430c0 2024-03-25T17:47:10.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.056+0000 7f845e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:47:10.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.056+0000 7f845e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:10.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.056+0000 7f845e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:10.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.056+0000 7f845e7fc640 10 monclient(hunting): my global_id is 4570 2024-03-25T17:47:10.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.056+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8448003520 con 0x7f84601430c0 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127972386 0 0) 0x7f8448003520 con 0x7f84601430c0 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 >> v1:172.21.15.44:6790/0 conn(0x7f84600a98f0 legacy=0x7f84600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 >> v1:172.21.15.44:6789/0 conn(0x7f846013f710 legacy=0x7f84601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 1 monclient: found mon.noname-c 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84600b44c0 con 0x7f84601430c0 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:47:10.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8458001a10 con 0x7f84601430c0 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f845e7fc640 10 monclient: dump: 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f8466255640 5 monclient: authenticate success, global_id 4570 2024-03-25T17:47:10.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.066+0000 7f8466255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.069+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8458001be0 con 0x7f84601430c0 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.069+0000 7f845e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.069+0000 7f845e7fc640 1 -- 172.21.15.44:0/3898207204 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8458002030 con 0x7f84601430c0 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.069+0000 7f8466255640 10 monclient: get_monmap_and_config success 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.069+0000 7f8466255640 4 set_mon_vals no callback set 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.069+0000 7f845e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.069+0000 7f845e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.069+0000 7f845e7fc640 10 monclient: dump: 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.070 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:10.071 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:10.071 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:10.071 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:10.071 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:10.071 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.070+0000 7f8466255640 10 monclient: shutdown 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.070+0000 7f8466255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.070+0000 7f8466255640 1 -- 172.21.15.44:0/3898207204 >> v1:172.21.15.111:6789/0 conn(0x7f84601430c0 legacy=0x7f846014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.070+0000 7f8466255640 1 -- 172.21.15.44:0/3898207204 shutdown_connections 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.070+0000 7f8466255640 1 -- 172.21.15.44:0/3898207204 >> 172.21.15.44:0/3898207204 conn(0x7f84600b7ea0 msgr2=0x7f84600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.070+0000 7f8466255640 1 -- 172.21.15.44:0/3898207204 shutdown_connections 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.071+0000 7f8466255640 1 -- 172.21.15.44:0/3898207204 wait complete. 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.071+0000 7f8466255640 10 monclient: build_initial_monmap 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.071+0000 7f8466255640 10 monclient: monmap: 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:10.072 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:47:10.073 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:10.073 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:47:10.073 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:47:10.073 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.072+0000 7f8466255640 1 librados: starting msgr at 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.072+0000 7f8466255640 1 librados: starting objecter 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.072+0000 7f8466255640 1 Processor -- start 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.072+0000 7f8466255640 1 -- start start 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.072+0000 7f8466255640 1 librados: setting wanted keys 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.072+0000 7f8466255640 1 librados: calling monclient init 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.072+0000 7f8466255640 10 monclient: init 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:10.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:47:10.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 10 monclient(hunting): picked mon.noname-b con 0x7f84600a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:10.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 10 monclient(hunting): picked mon.noname-a con 0x7f846013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:10.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 10 monclient(hunting): picked mon.noname-c con 0x7f84601430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:10.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84601dac00 con 0x7f846013f710 2024-03-25T17:47:10.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84600abc30 con 0x7f84600a98f0 2024-03-25T17:47:10.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84601d41e0 con 0x7f84601430c0 2024-03-25T17:47:10.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:10.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.073+0000 7f8466255640 10 monclient(hunting): authenticate will time out at 6433.301247s 2024-03-25T17:47:10.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.084+0000 7f845effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f846013f710 0x7f84601db0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53382/0 (socket says 172.21.15.44:53382) 2024-03-25T17:47:10.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.084+0000 7f845effd640 1 -- 172.21.15.44:0/2302494096 learned_addr learned my addr 172.21.15.44:0/2302494096 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:10.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.086+0000 7f8447fff640 10 client.?.objecter ms_handle_connect 0x7f846013f710 2024-03-25T17:47:10.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.086+0000 7f8447fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:10.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.092+0000 7f8447fff640 10 client.?.objecter ms_handle_connect 0x7f84601430c0 2024-03-25T17:47:10.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.092+0000 7f8447fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:10.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.097+0000 7f8447fff640 10 client.?.objecter ms_handle_connect 0x7f84600a98f0 2024-03-25T17:47:10.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.098+0000 7f8447fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:10.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.100+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063985022 0 0) 0x7f84601dac00 con 0x7f846013f710 2024-03-25T17:47:10.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.100+0000 7f8447fff640 10 monclient(hunting): _negotiate 2024-03-25T17:47:10.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.100+0000 7f8447fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:10.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.100+0000 7f8447fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:10.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.100+0000 7f8447fff640 10 monclient(hunting): my global_id is 4590 2024-03-25T17:47:10.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.100+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c004eb0 con 0x7f846013f710 2024-03-25T17:47:10.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.101+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845126559 0 0) 0x7f84601d41e0 con 0x7f84601430c0 2024-03-25T17:47:10.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.101+0000 7f8447fff640 10 monclient(hunting): _negotiate 2024-03-25T17:47:10.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.101+0000 7f8447fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:10.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.101+0000 7f8447fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:10.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.101+0000 7f8447fff640 10 monclient(hunting): my global_id is 4580 2024-03-25T17:47:10.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.101+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84601dac00 con 0x7f84601430c0 2024-03-25T17:47:10.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.105+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347228523 0 0) 0x7f84600abc30 con 0x7f84600a98f0 2024-03-25T17:47:10.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.105+0000 7f8447fff640 10 monclient(hunting): _negotiate 2024-03-25T17:47:10.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.105+0000 7f8447fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:10.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.105+0000 7f8447fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:10.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.105+0000 7f8447fff640 10 monclient(hunting): my global_id is 4573 2024-03-25T17:47:10.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.105+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84601d41e0 con 0x7f84600a98f0 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.114+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1705417948 0 0) 0x7f844c004eb0 con 0x7f846013f710 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.114+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84600abc30 con 0x7f846013f710 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.114+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8454004080 con 0x7f846013f710 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.114+0000 7f8447fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.114+0000 7f8447fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.114+0000 7f8447fff640 10 monclient(hunting): dump: 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:10.115 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:10.116 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:10.116 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:10.116 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:10.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.114+0000 7f8447fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:47:10.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.123+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1174642049 0 0) 0x7f84601dac00 con 0x7f84601430c0 2024-03-25T17:47:10.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.123+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f844c004eb0 con 0x7f84601430c0 2024-03-25T17:47:10.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.123+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8458001ef0 con 0x7f84601430c0 2024-03-25T17:47:10.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.123+0000 7f8447fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:10.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.123+0000 7f8447fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.123+0000 7f8447fff640 10 monclient(hunting): dump: 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:10.124 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 377586826 0 0) 0x7f84600abc30 con 0x7f846013f710 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 >> v1:172.21.15.44:6790/0 conn(0x7f84601430c0 legacy=0x7f84601d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 >> v1:172.21.15.111:6789/0 conn(0x7f84600a98f0 legacy=0x7f8460137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 10 monclient: _finish_hunting 0 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 1 monclient: found mon.a 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84601d4370 con 0x7f846013f710 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 10 monclient: _finish_auth 0 2024-03-25T17:47:10.137 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 10 monclient: _check_auth_tickets 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8447fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 5 monclient: authenticate success, global_id 4590 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 10 monclient: _renew_subs 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84601db7f0 con 0x7f846013f710 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 10 monclient: _renew_subs 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84601d50a0 con 0x7f846013f710 2024-03-25T17:47:10.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.136+0000 7f8466255640 1 librados: init done 2024-03-25T17:47:10.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.138+0000 7f8466255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:10.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.138+0000 7f8466255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:10.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.138+0000 7f8466255640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:10.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.138+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84601db9e0 con 0x7f846013f710 2024-03-25T17:47:10.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.463+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8454000ed0 con 0x7f846013f710 2024-03-25T17:47:10.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.463+0000 7f8447fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:10.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.463+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f84540012a0 con 0x7f846013f710 2024-03-25T17:47:10.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.463+0000 7f8447fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:10.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.463+0000 7f8447fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:47:10.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.463+0000 7f8447fff640 10 monclient: dump: 2024-03-25T17:47:10.464 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:10.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.463+0000 7f845dffb640 4 set_mon_vals no callback set 2024-03-25T17:47:10.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.464+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f84540014c0 con 0x7f846013f710 2024-03-25T17:47:10.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.465+0000 7f845f7fe640 10 monclient: get_auth_request con 0x7f844c06df10 auth_method 0 2024-03-25T17:47:10.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.465+0000 7f8447fff640 10 client.4590.objecter ms_handle_connect 0x7f844c06df10 2024-03-25T17:47:10.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.471+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f8454054770 con 0x7f846013f710 2024-03-25T17:47:10.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.471+0000 7f8447fff640 10 client.4590.objecter ms_dispatch 0x7f8460000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:47:10.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.471+0000 7f8447fff640 3 client.4590.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:47:10.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.471+0000 7f8447fff640 3 client.4590.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:47:10.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.471+0000 7f8447fff640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-25T17:47:10.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.471+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8454050280 con 0x7f846013f710 2024-03-25T17:47:10.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.471+0000 7f8447fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:10.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.471+0000 7f8447fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:47:10.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.701+0000 7f8466255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:10.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.701+0000 7f8466255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:10.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.701+0000 7f8466255640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:10.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.701+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84600627b0 con 0x7f846013f710 2024-03-25T17:47:10.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.745+0000 7f8447fff640 1 -- 172.21.15.44:0/2302494096 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f84540521b0 con 0x7f846013f710 2024-03-25T17:47:10.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.745+0000 7f8447fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:10.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.745+0000 7f8447fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:47:10.746 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:47:10.746 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 10 librados: watch_flush enter 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 10 librados: watch_flush exit 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 10 client.4590.objecter successfully canceled tick 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 >> v1:172.21.15.44:6800/3257043132 conn(0x7f844c06df10 legacy=0x7f844c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 10 monclient: shutdown 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 >> v1:172.21.15.44:6789/0 conn(0x7f846013f710 legacy=0x7f84601db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 shutdown_connections 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 >> 172.21.15.44:0/2302494096 conn(0x7f84600b7ea0 msgr2=0x7f8460074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 shutdown_connections 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 1 -- 172.21.15.44:0/2302494096 wait complete. 2024-03-25T17:47:10.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:10.748+0000 7f8466255640 1 librados: shutdown 2024-03-25T17:47:10.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:47:17.565 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.683+0000 7f1e563d7640 10 monclient: get_monmap_and_config 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.683+0000 7f1e563d7640 10 monclient: build_initial_monmap 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.683+0000 7f1e563d7640 10 monclient: monmap: 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:47:17.684386+0000 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:47:17.684386+0000 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:17.684 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:47:17.685 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:47:17.685 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.684+0000 7f1e563d7640 1 Processor -- start 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.684+0000 7f1e563d7640 1 -- start start 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.684+0000 7f1e563d7640 10 monclient: init 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 10 monclient(hunting): picked mon.noname-b con 0x7f1e500a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 10 monclient(hunting): picked mon.noname-c con 0x7f1e5013f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 10 monclient(hunting): picked mon.noname-a con 0x7f1e501430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e50142520 con 0x7f1e501430c0 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e500b28a0 con 0x7f1e5013f710 2024-03-25T17:47:17.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e500b2a30 con 0x7f1e500a98f0 2024-03-25T17:47:17.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:17.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.685+0000 7f1e563d7640 10 monclient(hunting): authenticate will time out at 6440.913040s 2024-03-25T17:47:17.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.698+0000 7f1e4ffff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f1e500a98f0 0x7f1e500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:54948/0 (socket says 172.21.15.44:54948) 2024-03-25T17:47:17.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.698+0000 7f1e4ffff640 1 -- 172.21.15.44:0/1198918867 learned_addr learned my addr 172.21.15.44:0/1198918867 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:17.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.706+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855392783 0 0) 0x7f1e500b28a0 con 0x7f1e5013f710 2024-03-25T17:47:17.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.706+0000 7f1e4effd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:17.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.706+0000 7f1e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:17.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.706+0000 7f1e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:17.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.706+0000 7f1e4effd640 10 monclient(hunting): my global_id is 4576 2024-03-25T17:47:17.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.706+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e38003520 con 0x7f1e5013f710 2024-03-25T17:47:17.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.716+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322381632 0 0) 0x7f1e500b2a30 con 0x7f1e500a98f0 2024-03-25T17:47:17.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.716+0000 7f1e4effd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:17.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.716+0000 7f1e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:17.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.716+0000 7f1e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:17.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.716+0000 7f1e4effd640 10 monclient(hunting): my global_id is 4583 2024-03-25T17:47:17.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.716+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e500b28a0 con 0x7f1e500a98f0 2024-03-25T17:47:17.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.719+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147245640 0 0) 0x7f1e50142520 con 0x7f1e501430c0 2024-03-25T17:47:17.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.719+0000 7f1e4effd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:17.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.719+0000 7f1e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:17.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.719+0000 7f1e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:17.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.719+0000 7f1e4effd640 10 monclient(hunting): my global_id is 4593 2024-03-25T17:47:17.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.719+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e500b2a30 con 0x7f1e501430c0 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2596533272 0 0) 0x7f1e38003520 con 0x7f1e5013f710 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 >> v1:172.21.15.44:6790/0 conn(0x7f1e500a98f0 legacy=0x7f1e500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 >> v1:172.21.15.44:6789/0 conn(0x7f1e501430c0 legacy=0x7f1e5014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 10 monclient: _finish_hunting 0 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 1 monclient: found mon.noname-c 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e500b44c0 con 0x7f1e5013f710 2024-03-25T17:47:17.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 10 monclient: _finish_auth 0 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 10 monclient: _check_auth_tickets 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1e44001690 con 0x7f1e5013f710 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e4effd640 10 monclient: dump: 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:17.725 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:17.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e563d7640 5 monclient: authenticate success, global_id 4576 2024-03-25T17:47:17.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.723+0000 7f1e563d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:17.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.729+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e440017f0 con 0x7f1e5013f710 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.729+0000 7f1e4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.729+0000 7f1e4effd640 1 -- 172.21.15.44:0/1198918867 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1e44001c00 con 0x7f1e5013f710 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.729+0000 7f1e563d7640 10 monclient: get_monmap_and_config success 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.729+0000 7f1e563d7640 4 set_mon_vals no callback set 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.729+0000 7f1e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.729+0000 7f1e4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.729+0000 7f1e4effd640 10 monclient: dump: 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:17.730 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:17.731 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:17.731 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:17.731 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:17.731 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:17.731 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:17.731 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.730+0000 7f1e563d7640 10 monclient: shutdown 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.730+0000 7f1e563d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.730+0000 7f1e563d7640 1 -- 172.21.15.44:0/1198918867 >> v1:172.21.15.111:6789/0 conn(0x7f1e5013f710 legacy=0x7f1e501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.730+0000 7f1e563d7640 1 -- 172.21.15.44:0/1198918867 shutdown_connections 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.730+0000 7f1e563d7640 1 -- 172.21.15.44:0/1198918867 >> 172.21.15.44:0/1198918867 conn(0x7f1e500b7ea0 msgr2=0x7f1e500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.730+0000 7f1e563d7640 1 -- 172.21.15.44:0/1198918867 shutdown_connections 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.730+0000 7f1e563d7640 1 -- 172.21.15.44:0/1198918867 wait complete. 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.731+0000 7f1e563d7640 10 monclient: build_initial_monmap 2024-03-25T17:47:17.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.731+0000 7f1e563d7640 10 monclient: monmap: 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.731+0000 7f1e563d7640 1 librados: starting msgr at 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.731+0000 7f1e563d7640 1 librados: starting objecter 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.731+0000 7f1e563d7640 1 Processor -- start 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 1 -- start start 2024-03-25T17:47:17.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 1 librados: setting wanted keys 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 1 librados: calling monclient init 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 10 monclient: init 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 10 monclient(hunting): picked mon.noname-a con 0x7f1e500a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 10 monclient(hunting): picked mon.noname-c con 0x7f1e5013f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 10 monclient(hunting): picked mon.noname-b con 0x7f1e501430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e50139870 con 0x7f1e500a98f0 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e501d51f0 con 0x7f1e501430c0 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e50138830 con 0x7f1e5013f710 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:17.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.732+0000 7f1e563d7640 10 monclient(hunting): authenticate will time out at 6440.960287s 2024-03-25T17:47:17.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.742+0000 7f1e54974640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1e501430c0 0x7f1e501e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:34930/0 (socket says 172.21.15.44:34930) 2024-03-25T17:47:17.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.742+0000 7f1e54974640 1 -- 172.21.15.44:0/2162972556 learned_addr learned my addr 172.21.15.44:0/2162972556 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:17.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.746+0000 7f1e4cff9640 10 client.?.objecter ms_handle_connect 0x7f1e501430c0 2024-03-25T17:47:17.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.746+0000 7f1e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:17.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.753+0000 7f1e4cff9640 10 client.?.objecter ms_handle_connect 0x7f1e5013f710 2024-03-25T17:47:17.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.753+0000 7f1e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:17.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.754+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053096289 0 0) 0x7f1e501d51f0 con 0x7f1e501430c0 2024-03-25T17:47:17.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.754+0000 7f1e4cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:47:17.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.754+0000 7f1e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:17.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.754+0000 7f1e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:17.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.754+0000 7f1e4cff9640 10 monclient(hunting): my global_id is 4579 2024-03-25T17:47:17.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.754+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e3c004eb0 con 0x7f1e501430c0 2024-03-25T17:47:17.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.763+0000 7f1e4cff9640 10 client.?.objecter ms_handle_connect 0x7f1e500a98f0 2024-03-25T17:47:17.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.763+0000 7f1e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:17.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.764+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662045117 0 0) 0x7f1e50138830 con 0x7f1e5013f710 2024-03-25T17:47:17.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.764+0000 7f1e4cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:47:17.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.764+0000 7f1e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:17.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.764+0000 7f1e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:17.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.764+0000 7f1e4cff9640 10 monclient(hunting): my global_id is 4586 2024-03-25T17:47:17.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.764+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e501d51f0 con 0x7f1e5013f710 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.767+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028472955 0 0) 0x7f1e3c004eb0 con 0x7f1e501430c0 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.767+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e50138830 con 0x7f1e501430c0 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.767+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1e48001c70 con 0x7f1e501430c0 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.767+0000 7f1e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.767+0000 7f1e4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.767+0000 7f1e4cff9640 10 monclient(hunting): dump: 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:17.768 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:17.769 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:17.769 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:17.769 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:17.769 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:17.769 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:17.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.767+0000 7f1e4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1856210312 0 0) 0x7f1e50138830 con 0x7f1e501430c0 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 >> v1:172.21.15.44:6790/0 conn(0x7f1e5013f710 legacy=0x7f1e501d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 >> v1:172.21.15.44:6789/0 conn(0x7f1e500a98f0 legacy=0x7f1e50138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 1 monclient: found mon.b 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e501d77a0 con 0x7f1e501430c0 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 10 monclient: _finish_auth 0 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:17.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 5 monclient: authenticate success, global_id 4579 2024-03-25T17:47:17.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 10 monclient: _renew_subs 2024-03-25T17:47:17.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:17.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e501d83d0 con 0x7f1e501430c0 2024-03-25T17:47:17.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:47:17.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 10 monclient: _renew_subs 2024-03-25T17:47:17.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:17.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e50138c80 con 0x7f1e501430c0 2024-03-25T17:47:17.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.772+0000 7f1e563d7640 1 librados: init done 2024-03-25T17:47:17.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.773+0000 7f1e563d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:17.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.773+0000 7f1e563d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:17.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.773+0000 7f1e563d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:17.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:17.773+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e500627b0 con 0x7f1e501430c0 2024-03-25T17:47:18.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.046+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e48003970 con 0x7f1e501430c0 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.046+0000 7f1e4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.046+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1e48003d00 con 0x7f1e501430c0 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.046+0000 7f1e4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.046+0000 7f1e4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.046+0000 7f1e4cff9640 10 monclient: dump: 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:18.047 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:18.048 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:18.048 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:18.048 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:18.048 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:18.048 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:18.048 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:18.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.046+0000 7f1e4e7fc640 4 set_mon_vals no callback set 2024-03-25T17:47:18.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.047+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f1e48003ee0 con 0x7f1e501430c0 2024-03-25T17:47:18.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.047+0000 7f1e4ffff640 10 monclient: get_auth_request con 0x7f1e3c06dd10 auth_method 0 2024-03-25T17:47:18.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.048+0000 7f1e4cff9640 10 client.4579.objecter ms_handle_connect 0x7f1e3c06dd10 2024-03-25T17:47:18.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.052+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f1e48054b30 con 0x7f1e501430c0 2024-03-25T17:47:18.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.052+0000 7f1e4cff9640 10 client.4579.objecter ms_dispatch 0x7f1e50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:47:18.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.052+0000 7f1e4cff9640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:47:18.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.052+0000 7f1e4cff9640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:47:18.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.053+0000 7f1e4cff9640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T17:47:18.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.053+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e48050640 con 0x7f1e501430c0 2024-03-25T17:47:18.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.053+0000 7f1e4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:18.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.053+0000 7f1e4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:47:18.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.322+0000 7f1e563d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:18.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.322+0000 7f1e563d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:18.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.322+0000 7f1e563d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:18.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.322+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e500b3e80 con 0x7f1e501430c0 2024-03-25T17:47:18.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.458+0000 7f1e4cff9640 1 -- 172.21.15.44:0/2162972556 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f1e48052570 con 0x7f1e501430c0 2024-03-25T17:47:18.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.458+0000 7f1e4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:18.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.458+0000 7f1e4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:47:18.459 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:47:18.459 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 10 librados: watch_flush enter 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 10 librados: watch_flush exit 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 10 client.4579.objecter successfully canceled tick 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 >> v1:172.21.15.44:6800/3257043132 conn(0x7f1e3c06dd10 legacy=0x7f1e3c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 10 monclient: shutdown 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 >> v1:172.21.15.111:6789/0 conn(0x7f1e501430c0 legacy=0x7f1e501e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 shutdown_connections 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 >> 172.21.15.44:0/2162972556 conn(0x7f1e500b7ea0 msgr2=0x7f1e50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 shutdown_connections 2024-03-25T17:47:18.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 1 -- 172.21.15.44:0/2162972556 wait complete. 2024-03-25T17:47:18.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:18.460+0000 7f1e563d7640 1 librados: shutdown 2024-03-25T17:47:18.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:47:18.680 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:36.036 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:18.744 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:36.099 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:18.778 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:36.133 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:18.783 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:36.139 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:18.799 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:36.155 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:18.803 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:36.123 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:18.807 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:36.127 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:18.834 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:36.154 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:18.838 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:36.194 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:25.279 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.394+0000 7fa78a5af640 10 monclient: get_monmap_and_config 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.394+0000 7fa78a5af640 10 monclient: build_initial_monmap 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.394+0000 7fa78a5af640 10 monclient: monmap: 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:47:25.395674+0000 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:47:25.395674+0000 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:25.395 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:47:25.396 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:47:25.396 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:25.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.395+0000 7fa78a5af640 1 Processor -- start 2024-03-25T17:47:25.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.395+0000 7fa78a5af640 1 -- start start 2024-03-25T17:47:25.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.395+0000 7fa78a5af640 10 monclient: init 2024-03-25T17:47:25.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:25.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:47:25.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 10 monclient(hunting): picked mon.noname-c con 0x7fa78406f360 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:25.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 10 monclient(hunting): picked mon.noname-a con 0x7fa78406f740 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:25.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 10 monclient(hunting): picked mon.noname-b con 0x7fa784142e30 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:25.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa784071d20 con 0x7fa78406f740 2024-03-25T17:47:25.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa784071340 con 0x7fa78406f360 2024-03-25T17:47:25.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa784071620 con 0x7fa784142e30 2024-03-25T17:47:25.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:25.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.396+0000 7fa78a5af640 10 monclient(hunting): authenticate will time out at 6448.624398s 2024-03-25T17:47:25.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.400+0000 7fa783fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa78406f360 0x7fa784142590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:58374/0 (socket says 172.21.15.44:58374) 2024-03-25T17:47:25.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.400+0000 7fa783fff640 1 -- 172.21.15.44:0/251982178 learned_addr learned my addr 172.21.15.44:0/251982178 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:25.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.415+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516062496 0 0) 0x7fa784071340 con 0x7fa78406f360 2024-03-25T17:47:25.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.415+0000 7fa782ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:25.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.415+0000 7fa782ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:25.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.415+0000 7fa782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:25.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.415+0000 7fa782ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T17:47:25.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.415+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c005730 con 0x7fa78406f360 2024-03-25T17:47:25.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.426+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561227591 0 0) 0x7fa784071620 con 0x7fa784142e30 2024-03-25T17:47:25.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.426+0000 7fa782ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:25.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.426+0000 7fa782ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:25.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.426+0000 7fa782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:25.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.426+0000 7fa782ffd640 10 monclient(hunting): my global_id is 4589 2024-03-25T17:47:25.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.426+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa784071340 con 0x7fa784142e30 2024-03-25T17:47:25.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.433+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553815856 0 0) 0x7fa784071d20 con 0x7fa78406f740 2024-03-25T17:47:25.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.433+0000 7fa782ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:25.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.433+0000 7fa782ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:25.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.433+0000 7fa782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:25.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.433+0000 7fa782ffd640 10 monclient(hunting): my global_id is 4599 2024-03-25T17:47:25.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.433+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa784071620 con 0x7fa78406f740 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293720317 0 0) 0x7fa76c005730 con 0x7fa78406f360 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 >> v1:172.21.15.44:6790/0 conn(0x7fa784142e30 legacy=0x7fa78414a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 >> v1:172.21.15.44:6789/0 conn(0x7fa78406f740 legacy=0x7fa784072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 1 monclient: found mon.noname-c 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa784071900 con 0x7fa78406f360 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 10 monclient: _finish_auth 0 2024-03-25T17:47:25.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa774001690 con 0x7fa78406f360 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa782ffd640 10 monclient: dump: 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:25.437 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:25.438 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:25.438 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:25.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa78a5af640 5 monclient: authenticate success, global_id 4582 2024-03-25T17:47:25.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.435+0000 7fa78a5af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:25.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.438+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7740017f0 con 0x7fa78406f360 2024-03-25T17:47:25.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.438+0000 7fa782ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:25.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.438+0000 7fa782ffd640 1 -- 172.21.15.44:0/251982178 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa774001b80 con 0x7fa78406f360 2024-03-25T17:47:25.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.438+0000 7fa78a5af640 10 monclient: get_monmap_and_config success 2024-03-25T17:47:25.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.438+0000 7fa78a5af640 4 set_mon_vals no callback set 2024-03-25T17:47:25.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.438+0000 7fa782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.438+0000 7fa782ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.438+0000 7fa782ffd640 10 monclient: dump: 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:25.440 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:25.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.439+0000 7fa78a5af640 10 monclient: shutdown 2024-03-25T17:47:25.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.439+0000 7fa78a5af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:25.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.439+0000 7fa78a5af640 1 -- 172.21.15.44:0/251982178 >> v1:172.21.15.111:6789/0 conn(0x7fa78406f360 legacy=0x7fa784142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:25.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.439+0000 7fa78a5af640 1 -- 172.21.15.44:0/251982178 shutdown_connections 2024-03-25T17:47:25.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.439+0000 7fa78a5af640 1 -- 172.21.15.44:0/251982178 >> 172.21.15.44:0/251982178 conn(0x7fa7840b4b80 msgr2=0x7fa7840b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.439+0000 7fa78a5af640 1 -- 172.21.15.44:0/251982178 shutdown_connections 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.439+0000 7fa78a5af640 1 -- 172.21.15.44:0/251982178 wait complete. 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.440+0000 7fa78a5af640 10 monclient: build_initial_monmap 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.440+0000 7fa78a5af640 10 monclient: monmap: 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:47:25.443 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.440+0000 7fa78a5af640 1 librados: starting msgr at 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.440+0000 7fa78a5af640 1 librados: starting objecter 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 1 Processor -- start 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 1 -- start start 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 1 librados: setting wanted keys 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 1 librados: calling monclient init 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 10 monclient: init 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 10 monclient(hunting): picked mon.noname-a con 0x7fa78406f360 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 10 monclient(hunting): picked mon.noname-c con 0x7fa78406f740 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.441+0000 7fa78a5af640 10 monclient(hunting): picked mon.noname-b con 0x7fa784142e30 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.442+0000 7fa78a5af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7841399f0 con 0x7fa78406f360 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.442+0000 7fa78a5af640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78413d970 con 0x7fa784142e30 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.442+0000 7fa78a5af640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78413a890 con 0x7fa78406f740 2024-03-25T17:47:25.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.442+0000 7fa78a5af640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:25.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.442+0000 7fa78a5af640 10 monclient(hunting): authenticate will time out at 6448.669732s 2024-03-25T17:47:25.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.452+0000 7fa783fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa78406f360 0x7fa7841392e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52806/0 (socket says 172.21.15.44:52806) 2024-03-25T17:47:25.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.452+0000 7fa783fff640 1 -- 172.21.15.44:0/400130529 learned_addr learned my addr 172.21.15.44:0/400130529 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:25.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.455+0000 7fa780ff9640 10 client.?.objecter ms_handle_connect 0x7fa78406f360 2024-03-25T17:47:25.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.455+0000 7fa780ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:25.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.456+0000 7fa780ff9640 10 client.?.objecter ms_handle_connect 0x7fa784142e30 2024-03-25T17:47:25.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.456+0000 7fa780ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:25.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.459+0000 7fa780ff9640 10 client.?.objecter ms_handle_connect 0x7fa78406f740 2024-03-25T17:47:25.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.459+0000 7fa780ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:25.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.472+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844480984 0 0) 0x7fa7841399f0 con 0x7fa78406f360 2024-03-25T17:47:25.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.472+0000 7fa780ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:47:25.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.472+0000 7fa780ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:25.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.472+0000 7fa780ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:25.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.472+0000 7fa780ff9640 10 monclient(hunting): my global_id is 4602 2024-03-25T17:47:25.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.472+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa770004eb0 con 0x7fa78406f360 2024-03-25T17:47:25.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.485+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797297822 0 0) 0x7fa78413d970 con 0x7fa784142e30 2024-03-25T17:47:25.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.485+0000 7fa780ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:47:25.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.485+0000 7fa780ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:25.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.485+0000 7fa780ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:25.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.485+0000 7fa780ff9640 10 monclient(hunting): my global_id is 4585 2024-03-25T17:47:25.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.485+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7841399f0 con 0x7fa784142e30 2024-03-25T17:47:25.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.486+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3231633260 0 0) 0x7fa770004eb0 con 0x7fa78406f360 2024-03-25T17:47:25.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.486+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa78413d970 con 0x7fa78406f360 2024-03-25T17:47:25.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.486+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa774001b00 con 0x7fa78406f360 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.486+0000 7fa780ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.486+0000 7fa780ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.486+0000 7fa780ff9640 10 monclient(hunting): dump: 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:25.488 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.486+0000 7fa780ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.488+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337964564 0 0) 0x7fa78413a890 con 0x7fa78406f740 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.488+0000 7fa780ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.488+0000 7fa780ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:25.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.488+0000 7fa780ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:25.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.488+0000 7fa780ff9640 10 monclient(hunting): my global_id is 4592 2024-03-25T17:47:25.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.488+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa770004eb0 con 0x7fa78406f740 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.503+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1033975898 0 0) 0x7fa770004eb0 con 0x7fa78406f740 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.504+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa78413a890 con 0x7fa78406f740 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.504+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa778001830 con 0x7fa78406f740 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.504+0000 7fa780ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.504+0000 7fa780ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.504+0000 7fa780ff9640 10 monclient(hunting): dump: 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:25.504 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.505 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.505 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:25.505 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:25.505 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:25.505 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:25.505 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:25.505 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3849282050 0 0) 0x7fa78413d970 con 0x7fa78406f360 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 >> v1:172.21.15.44:6790/0 conn(0x7fa78406f740 legacy=0x7fa78413a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 >> v1:172.21.15.111:6789/0 conn(0x7fa784142e30 legacy=0x7fa7841e2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 1 monclient: found mon.a 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa784141800 con 0x7fa78406f360 2024-03-25T17:47:25.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 10 monclient: _finish_auth 0 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa780ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 5 monclient: authenticate success, global_id 4602 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 10 monclient: _renew_subs 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa784141ad0 con 0x7fa78406f360 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 10 monclient: _renew_subs 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa784137bf0 con 0x7fa78406f360 2024-03-25T17:47:25.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.509+0000 7fa78a5af640 1 librados: init done 2024-03-25T17:47:25.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.510+0000 7fa78a5af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:25.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.510+0000 7fa78a5af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:25.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.510+0000 7fa78a5af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:25.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.510+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7840627b0 con 0x7fa78406f360 2024-03-25T17:47:25.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.819+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa774004a90 con 0x7fa78406f360 2024-03-25T17:47:25.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.819+0000 7fa780ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.819+0000 7fa7827fc640 4 set_mon_vals no callback set 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.819+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa774004da0 con 0x7fa78406f360 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.819+0000 7fa780ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.819+0000 7fa780ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.819+0000 7fa780ff9640 10 monclient: dump: 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:25.820 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:25.821 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:25.821 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:25.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.820+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fa77401ad70 con 0x7fa78406f360 2024-03-25T17:47:25.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.820+0000 7fa7837fe640 10 monclient: get_auth_request con 0x7fa77006df70 auth_method 0 2024-03-25T17:47:25.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.820+0000 7fa780ff9640 10 client.4602.objecter ms_handle_connect 0x7fa77006df70 2024-03-25T17:47:25.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.826+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fa7740549d0 con 0x7fa78406f360 2024-03-25T17:47:25.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.826+0000 7fa780ff9640 10 client.4602.objecter ms_dispatch 0x7fa784000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:47:25.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.826+0000 7fa780ff9640 3 client.4602.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:47:25.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.826+0000 7fa780ff9640 3 client.4602.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:47:25.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.826+0000 7fa780ff9640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-25T17:47:25.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.826+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7740504e0 con 0x7fa78406f360 2024-03-25T17:47:25.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.827+0000 7fa780ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:25.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:25.827+0000 7fa780ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:47:26.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.080+0000 7fa78a5af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:26.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.080+0000 7fa78a5af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:26.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.080+0000 7fa78a5af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:47:26.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.080+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7840b5440 con 0x7fa78406f360 2024-03-25T17:47:26.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.112+0000 7fa780ff9640 1 -- 172.21.15.44:0/400130529 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fa774052410 con 0x7fa78406f360 2024-03-25T17:47:26.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.112+0000 7fa780ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:26.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.112+0000 7fa780ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:47:26.113 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:47:26.113 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:47:26.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 10 librados: watch_flush enter 2024-03-25T17:47:26.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 10 librados: watch_flush exit 2024-03-25T17:47:26.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-25T17:47:26.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 10 client.4602.objecter successfully canceled tick 2024-03-25T17:47:26.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 >> v1:172.21.15.44:6800/3257043132 conn(0x7fa77006df70 legacy=0x7fa770070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:26.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 10 monclient: shutdown 2024-03-25T17:47:26.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:26.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 >> v1:172.21.15.44:6789/0 conn(0x7fa78406f360 legacy=0x7fa7841392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:26.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 shutdown_connections 2024-03-25T17:47:26.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.115+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 >> 172.21.15.44:0/400130529 conn(0x7fa7840b4b80 msgr2=0x7fa7840b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:26.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.116+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 shutdown_connections 2024-03-25T17:47:26.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.116+0000 7fa78a5af640 1 -- 172.21.15.44:0/400130529 wait complete. 2024-03-25T17:47:26.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:26.116+0000 7fa78a5af640 1 librados: shutdown 2024-03-25T17:47:26.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:47:33.022 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:47:33.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.141+0000 7f112415d640 10 monclient: get_monmap_and_config 2024-03-25T17:47:33.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.141+0000 7f112415d640 10 monclient: build_initial_monmap 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.141+0000 7f112415d640 10 monclient: monmap: 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:47:33.141992+0000 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:47:33.141992+0000 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:47:33.142 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.141+0000 7f112415d640 1 Processor -- start 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 1 -- start start 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 10 monclient: init 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 10 monclient(hunting): picked mon.noname-b con 0x7f111c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 10 monclient(hunting): picked mon.noname-c con 0x7f111c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 10 monclient(hunting): picked mon.noname-a con 0x7f111c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c142520 con 0x7f111c1430c0 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c0b28a0 con 0x7f111c13f710 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c0b2a30 con 0x7f111c0a98f0 2024-03-25T17:47:33.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:33.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.142+0000 7f112415d640 10 monclient(hunting): authenticate will time out at 6456.370588s 2024-03-25T17:47:33.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.146+0000 7f1121ef9640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f111c0a98f0 0x7f111c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:36942/0 (socket says 172.21.15.44:36942) 2024-03-25T17:47:33.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.146+0000 7f1121ef9640 1 -- 172.21.15.44:0/848135487 learned_addr learned my addr 172.21.15.44:0/848135487 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:33.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.155+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765351012 0 0) 0x7f111c0b2a30 con 0x7f111c0a98f0 2024-03-25T17:47:33.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.155+0000 7f1120ef7640 10 monclient(hunting): _negotiate 2024-03-25T17:47:33.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.155+0000 7f1120ef7640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:33.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.155+0000 7f1120ef7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:33.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.155+0000 7f1120ef7640 10 monclient(hunting): my global_id is 4595 2024-03-25T17:47:33.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.155+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1108003520 con 0x7f111c0a98f0 2024-03-25T17:47:33.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.166+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910797819 0 0) 0x7f1108003520 con 0x7f111c0a98f0 2024-03-25T17:47:33.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.166+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 >> v1:172.21.15.111:6789/0 conn(0x7f111c13f710 legacy=0x7f111c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:33.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.166+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 >> v1:172.21.15.44:6789/0 conn(0x7f111c1430c0 legacy=0x7f111c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:33.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.166+0000 7f1120ef7640 10 monclient: _finish_hunting 0 2024-03-25T17:47:33.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.166+0000 7f1120ef7640 1 monclient: found mon.noname-b 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.166+0000 7f1120ef7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0b44c0 con 0x7f111c0a98f0 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 10 monclient: _finish_auth 0 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 10 monclient: _check_auth_tickets 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1118001690 con 0x7f111c0a98f0 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f1120ef7640 10 monclient: dump: 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:33.168 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:33.169 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:33.169 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:33.169 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:33.169 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:33.169 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:33.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f112415d640 5 monclient: authenticate success, global_id 4595 2024-03-25T17:47:33.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.167+0000 7f112415d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.180+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11180017f0 con 0x7f111c0a98f0 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.180+0000 7f1120ef7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.180+0000 7f1120ef7640 1 -- 172.21.15.44:0/848135487 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1118001c00 con 0x7f111c0a98f0 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.180+0000 7f112415d640 10 monclient: get_monmap_and_config success 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.180+0000 7f112415d640 4 set_mon_vals no callback set 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.180+0000 7f1120ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.180+0000 7f1120ef7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.180+0000 7f1120ef7640 10 monclient: dump: 2024-03-25T17:47:33.181 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:33.182 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:33.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.181+0000 7f112415d640 10 monclient: shutdown 2024-03-25T17:47:33.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.181+0000 7f112415d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.181+0000 7f112415d640 1 -- 172.21.15.44:0/848135487 >> v1:172.21.15.44:6790/0 conn(0x7f111c0a98f0 legacy=0x7f111c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.181+0000 7f112415d640 1 -- 172.21.15.44:0/848135487 shutdown_connections 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.181+0000 7f112415d640 1 -- 172.21.15.44:0/848135487 >> 172.21.15.44:0/848135487 conn(0x7f111c0b7ea0 msgr2=0x7f111c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.181+0000 7f112415d640 1 -- 172.21.15.44:0/848135487 shutdown_connections 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.182+0000 7f112415d640 1 -- 172.21.15.44:0/848135487 wait complete. 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.182+0000 7f112415d640 10 monclient: build_initial_monmap 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.182+0000 7f112415d640 10 monclient: monmap: 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:33.184 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.183+0000 7f112415d640 1 librados: starting msgr at 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.183+0000 7f112415d640 1 librados: starting objecter 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.183+0000 7f112415d640 1 Processor -- start 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.183+0000 7f112415d640 1 -- start start 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.183+0000 7f112415d640 1 librados: setting wanted keys 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.183+0000 7f112415d640 1 librados: calling monclient init 2024-03-25T17:47:33.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.183+0000 7f112415d640 10 monclient: init 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 10 monclient(hunting): picked mon.noname-b con 0x7f111c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 10 monclient(hunting): picked mon.noname-a con 0x7f111c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 10 monclient(hunting): picked mon.noname-c con 0x7f111c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c139a40 con 0x7f111c13f710 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c133f20 con 0x7f111c0a98f0 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c0ab950 con 0x7f111c1430c0 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:33.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.184+0000 7f112415d640 10 monclient(hunting): authenticate will time out at 6456.412042s 2024-03-25T17:47:33.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.198+0000 7f1121ef9640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f111c0a98f0 0x7f111c138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:43628/0 (socket says 172.21.15.44:43628) 2024-03-25T17:47:33.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.198+0000 7f1121ef9640 1 -- 172.21.15.44:0/2355375598 learned_addr learned my addr 172.21.15.44:0/2355375598 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:33.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.200+0000 7f11127fc640 10 client.?.objecter ms_handle_connect 0x7f111c13f710 2024-03-25T17:47:33.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.200+0000 7f11127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:33.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.203+0000 7f11127fc640 10 client.?.objecter ms_handle_connect 0x7f111c0a98f0 2024-03-25T17:47:33.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.203+0000 7f11127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:33.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.207+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360204096 0 0) 0x7f111c133f20 con 0x7f111c0a98f0 2024-03-25T17:47:33.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.207+0000 7f11127fc640 10 monclient(hunting): _negotiate 2024-03-25T17:47:33.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.207+0000 7f11127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:33.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.207+0000 7f11127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:33.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.207+0000 7f11127fc640 10 monclient(hunting): my global_id is 4591 2024-03-25T17:47:33.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.207+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1100004eb0 con 0x7f111c0a98f0 2024-03-25T17:47:33.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.210+0000 7f11127fc640 10 client.?.objecter ms_handle_connect 0x7f111c1430c0 2024-03-25T17:47:33.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.210+0000 7f11127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:33.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.212+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205884458 0 0) 0x7f111c139a40 con 0x7f111c13f710 2024-03-25T17:47:33.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.212+0000 7f11127fc640 10 monclient(hunting): _negotiate 2024-03-25T17:47:33.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.212+0000 7f11127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:33.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.212+0000 7f11127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:33.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.212+0000 7f11127fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T17:47:33.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.212+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111c133f20 con 0x7f111c13f710 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.215+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2380154138 0 0) 0x7f1100004eb0 con 0x7f111c0a98f0 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.215+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f111c139a40 con 0x7f111c0a98f0 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.215+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1118001bc0 con 0x7f111c0a98f0 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.215+0000 7f11127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.215+0000 7f11127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.215+0000 7f11127fc640 10 monclient(hunting): dump: 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:33.216 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:33.217 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:33.217 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:33.217 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:33.217 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:33.217 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:33.217 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:33.217 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:33.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.215+0000 7f11127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:47:33.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.224+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2079187387 0 0) 0x7f111c139a40 con 0x7f111c0a98f0 2024-03-25T17:47:33.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.224+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 >> v1:172.21.15.44:6790/0 conn(0x7f111c1430c0 legacy=0x7f111c133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.224+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 >> v1:172.21.15.44:6789/0 conn(0x7f111c13f710 legacy=0x7f111c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.224+0000 7f11127fc640 10 monclient: _finish_hunting 0 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.224+0000 7f11127fc640 1 monclient: found mon.b 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f11127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f11127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0abc30 con 0x7f111c0a98f0 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f11127fc640 10 monclient: _finish_auth 0 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f11127fc640 10 monclient: _check_auth_tickets 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f11127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 5 monclient: authenticate success, global_id 4591 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 10 monclient: _renew_subs 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c0abf10 con 0x7f111c0a98f0 2024-03-25T17:47:33.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:47:33.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 10 monclient: _renew_subs 2024-03-25T17:47:33.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:33.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c139370 con 0x7f111c0a98f0 2024-03-25T17:47:33.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.225+0000 7f112415d640 1 librados: init done 2024-03-25T17:47:33.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.226+0000 7f112415d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:33.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.226+0000 7f112415d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:33.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.226+0000 7f112415d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:33.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.226+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c0627b0 con 0x7f111c0a98f0 2024-03-25T17:47:33.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.492+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1118004a90 con 0x7f111c0a98f0 2024-03-25T17:47:33.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.492+0000 7f11127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:33.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.492+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1118004da0 con 0x7f111c0a98f0 2024-03-25T17:47:33.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.492+0000 7f1113fff640 4 set_mon_vals no callback set 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.492+0000 7f11127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.492+0000 7f11127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.492+0000 7f11127fc640 10 monclient: dump: 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:33.494 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:33.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.493+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f111801ad70 con 0x7f111c0a98f0 2024-03-25T17:47:33.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.494+0000 7f11216f8640 10 monclient: get_auth_request con 0x7f110006dcc0 auth_method 0 2024-03-25T17:47:33.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.494+0000 7f11127fc640 10 client.4591.objecter ms_handle_connect 0x7f110006dcc0 2024-03-25T17:47:33.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.497+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f1118054930 con 0x7f111c0a98f0 2024-03-25T17:47:33.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.498+0000 7f11127fc640 10 client.4591.objecter ms_dispatch 0x7f111c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:47:33.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.498+0000 7f11127fc640 3 client.4591.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:47:33.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.498+0000 7f11127fc640 3 client.4591.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:47:33.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.498+0000 7f11127fc640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-25T17:47:33.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.498+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1118050270 con 0x7f111c0a98f0 2024-03-25T17:47:33.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.498+0000 7f11127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:33.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.498+0000 7f11127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:47:33.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.753+0000 7f112415d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:33.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.753+0000 7f112415d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:33.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.753+0000 7f112415d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:33.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.753+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f111c0b7180 con 0x7f111c0a98f0 2024-03-25T17:47:33.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.776+0000 7f11127fc640 1 -- 172.21.15.44:0/2355375598 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f1118052370 con 0x7f111c0a98f0 2024-03-25T17:47:33.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.776+0000 7f11127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:33.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.776+0000 7f11127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:47:33.776 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:47:33.777 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:47:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 10 librados: watch_flush enter 2024-03-25T17:47:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 10 librados: watch_flush exit 2024-03-25T17:47:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-25T17:47:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 10 client.4591.objecter successfully canceled tick 2024-03-25T17:47:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 >> v1:172.21.15.44:6800/3257043132 conn(0x7f110006dcc0 legacy=0x7f1100070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 10 monclient: shutdown 2024-03-25T17:47:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 >> v1:172.21.15.111:6789/0 conn(0x7f111c0a98f0 legacy=0x7f111c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 shutdown_connections 2024-03-25T17:47:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 >> 172.21.15.44:0/2355375598 conn(0x7f111c0b7ea0 msgr2=0x7f111c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 shutdown_connections 2024-03-25T17:47:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 1 -- 172.21.15.44:0/2355375598 wait complete. 2024-03-25T17:47:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:33.778+0000 7f112415d640 1 librados: shutdown 2024-03-25T17:47:33.792 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:47:38.843 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:56.198 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:38.853 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:56.208 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:38.886 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:56.242 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:38.903 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:56.223 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:38.911 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:56.230 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:38.923 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:56.279 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:38.926 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:56.282 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:38.939 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:56.259 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:38.948 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:56.303 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:40.598 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.708+0000 7fb252bd7640 10 monclient: get_monmap_and_config 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.708+0000 7fb252bd7640 10 monclient: build_initial_monmap 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.708+0000 7fb252bd7640 10 monclient: monmap: 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:47:40.709532+0000 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:47:40.709532+0000 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:40.709 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:47:40.710 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:47:40.710 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:40.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.709+0000 7fb252bd7640 1 Processor -- start 2024-03-25T17:47:40.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.709+0000 7fb252bd7640 1 -- start start 2024-03-25T17:47:40.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.709+0000 7fb252bd7640 10 monclient: init 2024-03-25T17:47:40.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:40.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:47:40.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 10 monclient(hunting): picked mon.noname-b con 0x7fb24c06c380 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:40.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 10 monclient(hunting): picked mon.noname-a con 0x7fb24c0b12d0 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:40.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 10 monclient(hunting): picked mon.noname-c con 0x7fb24c0aedf0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:40.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0b16f0 con 0x7fb24c0b12d0 2024-03-25T17:47:40.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0adfe0 con 0x7fb24c0aedf0 2024-03-25T17:47:40.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0add00 con 0x7fb24c06c380 2024-03-25T17:47:40.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:40.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.710+0000 7fb252bd7640 10 monclient(hunting): authenticate will time out at 6463.938571s 2024-03-25T17:47:40.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.719+0000 7fb251174640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb24c0aedf0 0x7fb24c14e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:55048/0 (socket says 172.21.15.44:55048) 2024-03-25T17:47:40.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.719+0000 7fb250973640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fb24c06c380 0x7fb24c0b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:49536/0 (socket says 172.21.15.44:49536) 2024-03-25T17:47:40.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.719+0000 7fb251174640 1 -- 172.21.15.44:0/2427980959 learned_addr learned my addr 172.21.15.44:0/2427980959 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:40.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.728+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095024189 0 0) 0x7fb24c0adfe0 con 0x7fb24c0aedf0 2024-03-25T17:47:40.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.728+0000 7fb24b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:47:40.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.728+0000 7fb24b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:40.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.728+0000 7fb24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:40.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.728+0000 7fb24b7fe640 10 monclient(hunting): my global_id is 4594 2024-03-25T17:47:40.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.728+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb23c003520 con 0x7fb24c0aedf0 2024-03-25T17:47:40.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.729+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825072785 0 0) 0x7fb24c0add00 con 0x7fb24c06c380 2024-03-25T17:47:40.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.729+0000 7fb24b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:47:40.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.729+0000 7fb24b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:40.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.729+0000 7fb24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:40.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.729+0000 7fb24b7fe640 10 monclient(hunting): my global_id is 4601 2024-03-25T17:47:40.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.729+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c0adfe0 con 0x7fb24c06c380 2024-03-25T17:47:40.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 213836524 0 0) 0x7fb24c0adfe0 con 0x7fb24c06c380 2024-03-25T17:47:40.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 >> v1:172.21.15.111:6789/0 conn(0x7fb24c0aedf0 legacy=0x7fb24c14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:40.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 >> v1:172.21.15.44:6789/0 conn(0x7fb24c0b12d0 legacy=0x7fb24c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:47:40.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 1 monclient: found mon.noname-b 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c0ae2c0 con 0x7fb24c06c380 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb240001670 con 0x7fb24c06c380 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb252bd7640 5 monclient: authenticate success, global_id 4601 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb252bd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb24b7fe640 10 monclient: dump: 2024-03-25T17:47:40.750 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:40.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.748+0000 7fb252bd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:40.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.776+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2400017d0 con 0x7fb24c06c380 2024-03-25T17:47:40.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.776+0000 7fb24b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.776+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2427980959 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb240001ba0 con 0x7fb24c06c380 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.776+0000 7fb252bd7640 10 monclient: get_monmap_and_config success 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.776+0000 7fb252bd7640 4 set_mon_vals no callback set 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.776+0000 7fb24b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.776+0000 7fb24b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.776+0000 7fb24b7fe640 10 monclient: dump: 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:40.777 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:40.778 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:40.778 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:40.778 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:40.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.777+0000 7fb252bd7640 10 monclient: shutdown 2024-03-25T17:47:40.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.777+0000 7fb252bd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:40.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.777+0000 7fb252bd7640 1 -- 172.21.15.44:0/2427980959 >> v1:172.21.15.44:6790/0 conn(0x7fb24c06c380 legacy=0x7fb24c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:40.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.777+0000 7fb252bd7640 1 -- 172.21.15.44:0/2427980959 shutdown_connections 2024-03-25T17:47:40.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.777+0000 7fb252bd7640 1 -- 172.21.15.44:0/2427980959 >> 172.21.15.44:0/2427980959 conn(0x7fb24c0b5d10 msgr2=0x7fb24c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:40.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.777+0000 7fb252bd7640 1 -- 172.21.15.44:0/2427980959 shutdown_connections 2024-03-25T17:47:40.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.777+0000 7fb252bd7640 1 -- 172.21.15.44:0/2427980959 wait complete. 2024-03-25T17:47:40.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.778+0000 7fb252bd7640 10 monclient: build_initial_monmap 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.778+0000 7fb252bd7640 10 monclient: monmap: 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.778+0000 7fb252bd7640 1 librados: starting msgr at 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.778+0000 7fb252bd7640 1 librados: starting objecter 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 1 Processor -- start 2024-03-25T17:47:40.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 1 -- start start 2024-03-25T17:47:40.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 1 librados: setting wanted keys 2024-03-25T17:47:40.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 1 librados: calling monclient init 2024-03-25T17:47:40.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 10 monclient: init 2024-03-25T17:47:40.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:40.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:47:40.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 10 monclient(hunting): picked mon.noname-c con 0x7fb24c06c380 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:40.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 10 monclient(hunting): picked mon.noname-a con 0x7fb24c0aedf0 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:40.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 10 monclient(hunting): picked mon.noname-b con 0x7fb24c0b12d0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:40.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c06c8e0 con 0x7fb24c0aedf0 2024-03-25T17:47:40.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.779+0000 7fb252bd7640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c1a5880 con 0x7fb24c0b12d0 2024-03-25T17:47:40.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.780+0000 7fb252bd7640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c1a6e00 con 0x7fb24c06c380 2024-03-25T17:47:40.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.780+0000 7fb252bd7640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:40.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.780+0000 7fb252bd7640 10 monclient(hunting): authenticate will time out at 6464.007696s 2024-03-25T17:47:40.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.791+0000 7fb250973640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fb24c06c380 0x7fb24c1d7220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:49588/0 (socket says 172.21.15.44:49588) 2024-03-25T17:47:40.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.791+0000 7fb250973640 1 -- 172.21.15.44:0/2737974713 learned_addr learned my addr 172.21.15.44:0/2737974713 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:40.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.792+0000 7fb2497fa640 10 client.?.objecter ms_handle_connect 0x7fb24c06c380 2024-03-25T17:47:40.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.792+0000 7fb2497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:40.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.794+0000 7fb2497fa640 10 client.?.objecter ms_handle_connect 0x7fb24c0aedf0 2024-03-25T17:47:40.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.794+0000 7fb2497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:40.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.802+0000 7fb2497fa640 10 client.?.objecter ms_handle_connect 0x7fb24c0b12d0 2024-03-25T17:47:40.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.802+0000 7fb2497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:40.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776008273 0 0) 0x7fb24c1a6e00 con 0x7fb24c06c380 2024-03-25T17:47:40.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 10 monclient(hunting): _negotiate 2024-03-25T17:47:40.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:40.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:40.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 10 monclient(hunting): my global_id is 4604 2024-03-25T17:47:40.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb230004eb0 con 0x7fb24c06c380 2024-03-25T17:47:40.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119067756 0 0) 0x7fb24c06c8e0 con 0x7fb24c0aedf0 2024-03-25T17:47:40.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 10 monclient(hunting): _negotiate 2024-03-25T17:47:40.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:40.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:40.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 10 monclient(hunting): my global_id is 4608 2024-03-25T17:47:40.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.803+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c1a6e00 con 0x7fb24c0aedf0 2024-03-25T17:47:40.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.810+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389368604 0 0) 0x7fb24c1a5880 con 0x7fb24c0b12d0 2024-03-25T17:47:40.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.810+0000 7fb2497fa640 10 monclient(hunting): _negotiate 2024-03-25T17:47:40.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.810+0000 7fb2497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:40.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.810+0000 7fb2497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:40.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.810+0000 7fb2497fa640 10 monclient(hunting): my global_id is 4597 2024-03-25T17:47:40.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.810+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c06c8e0 con 0x7fb24c0b12d0 2024-03-25T17:47:40.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.815+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 928966334 0 0) 0x7fb24c1a6e00 con 0x7fb24c0aedf0 2024-03-25T17:47:40.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.815+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb24c1a5880 con 0x7fb24c0aedf0 2024-03-25T17:47:40.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.815+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb238001770 con 0x7fb24c0aedf0 2024-03-25T17:47:40.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.815+0000 7fb2497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:40.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.815+0000 7fb2497fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:47:40.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.815+0000 7fb2497fa640 10 monclient(hunting): dump: 2024-03-25T17:47:40.816 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:40.816 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:40.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.815+0000 7fb2497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.823+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1056930654 0 0) 0x7fb230004eb0 con 0x7fb24c06c380 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.823+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb24c1a6e00 con 0x7fb24c06c380 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.823+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb240001ba0 con 0x7fb24c06c380 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.823+0000 7fb2497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.823+0000 7fb2497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.823+0000 7fb2497fa640 10 monclient(hunting): dump: 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:40.824 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:40.825 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:40.825 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:40.825 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:40.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.826+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500707113 0 0) 0x7fb24c06c8e0 con 0x7fb24c0b12d0 2024-03-25T17:47:40.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.826+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb230004eb0 con 0x7fb24c0b12d0 2024-03-25T17:47:40.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.826+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb244001cd0 con 0x7fb24c0b12d0 2024-03-25T17:47:40.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.826+0000 7fb2497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:40.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.826+0000 7fb2497fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:40.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.826+0000 7fb2497fa640 10 monclient(hunting): dump: 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:40.827 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3236951099 0 0) 0x7fb24c1a6e00 con 0x7fb24c06c380 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 >> v1:172.21.15.111:6789/0 conn(0x7fb24c0b12d0 legacy=0x7fb24c1c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 >> v1:172.21.15.44:6789/0 conn(0x7fb24c0aedf0 legacy=0x7fb24c1c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 10 monclient: _finish_hunting 0 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 1 monclient: found mon.c 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c1c6600 con 0x7fb24c06c380 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 10 monclient: _finish_auth 0 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 10 monclient: _check_auth_tickets 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb2497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 5 monclient: authenticate success, global_id 4604 2024-03-25T17:47:40.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 10 monclient: _renew_subs 2024-03-25T17:47:40.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:40.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c1d79f0 con 0x7fb24c06c380 2024-03-25T17:47:40.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:47:40.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 10 monclient: _renew_subs 2024-03-25T17:47:40.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:40.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c1a5210 con 0x7fb24c06c380 2024-03-25T17:47:40.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.831+0000 7fb252bd7640 1 librados: init done 2024-03-25T17:47:40.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.832+0000 7fb252bd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:40.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.832+0000 7fb252bd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:40.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.832+0000 7fb252bd7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:40.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.832+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c06d790 con 0x7fb24c06c380 2024-03-25T17:47:40.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.869+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb240004a90 con 0x7fb24c06c380 2024-03-25T17:47:40.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.869+0000 7fb2497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:40.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.869+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fb240004c00 con 0x7fb24c06c380 2024-03-25T17:47:40.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.869+0000 7fb2497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:40.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.869+0000 7fb2497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:40.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.869+0000 7fb2497fa640 10 monclient: dump: 2024-03-25T17:47:40.869 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:40.869 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:40.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.869+0000 7fb24affd640 4 set_mon_vals no callback set 2024-03-25T17:47:40.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.869+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fb24001abd0 con 0x7fb24c06c380 2024-03-25T17:47:40.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.870+0000 7fb24bfff640 10 monclient: get_auth_request con 0x7fb23006e0e0 auth_method 0 2024-03-25T17:47:40.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.870+0000 7fb2497fa640 10 client.4604.objecter ms_handle_connect 0x7fb23006e0e0 2024-03-25T17:47:40.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.873+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fb240054800 con 0x7fb24c06c380 2024-03-25T17:47:40.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.873+0000 7fb2497fa640 10 client.4604.objecter ms_dispatch 0x7fb24c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:47:40.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.873+0000 7fb2497fa640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:47:40.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.873+0000 7fb2497fa640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:47:40.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:40.873+0000 7fb2497fa640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T17:47:41.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.202+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb240050310 con 0x7fb24c06c380 2024-03-25T17:47:41.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.202+0000 7fb2497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:41.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.202+0000 7fb2497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:47:41.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.440+0000 7fb252bd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:41.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.440+0000 7fb252bd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:41.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.440+0000 7fb252bd7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:41.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.440+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb24c0627b0 con 0x7fb24c06c380 2024-03-25T17:47:41.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.468+0000 7fb2497fa640 1 -- 172.21.15.44:0/2737974713 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fb240052240 con 0x7fb24c06c380 2024-03-25T17:47:41.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.468+0000 7fb2497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:41.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.468+0000 7fb2497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:47:41.469 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:47:41.469 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:47:41.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 10 librados: watch_flush enter 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 10 librados: watch_flush exit 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 10 client.4604.objecter successfully canceled tick 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 >> v1:172.21.15.44:6800/3257043132 conn(0x7fb23006e0e0 legacy=0x7fb2300705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 10 monclient: shutdown 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 >> v1:172.21.15.44:6790/0 conn(0x7fb24c06c380 legacy=0x7fb24c1d7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 shutdown_connections 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 >> 172.21.15.44:0/2737974713 conn(0x7fb24c0b5d10 msgr2=0x7fb24c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 shutdown_connections 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 1 -- 172.21.15.44:0/2737974713 wait complete. 2024-03-25T17:47:41.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:41.471+0000 7fb252bd7640 1 librados: shutdown 2024-03-25T17:47:41.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:47:48.290 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:47:48.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.408+0000 7fbcf6732640 10 monclient: get_monmap_and_config 2024-03-25T17:47:48.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.408+0000 7fbcf6732640 10 monclient: build_initial_monmap 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.408+0000 7fbcf6732640 10 monclient: monmap: 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:47:48.409005+0000 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:47:48.409005+0000 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:47:48.409 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:48.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.408+0000 7fbcf6732640 1 Processor -- start 2024-03-25T17:47:48.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.408+0000 7fbcf6732640 1 -- start start 2024-03-25T17:47:48.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.408+0000 7fbcf6732640 10 monclient: init 2024-03-25T17:47:48.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf00a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf01430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf0142520 con 0x7fbcf013f710 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf00b28a0 con 0x7fbcf01430c0 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf00b2a30 con 0x7fbcf00a98f0 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:48.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.409+0000 7fbcf6732640 10 monclient(hunting): authenticate will time out at 6471.637302s 2024-03-25T17:47:48.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.421+0000 7fbceffff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fbcf00a98f0 0x7fbcf00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:54466/0 (socket says 172.21.15.44:54466) 2024-03-25T17:47:48.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.421+0000 7fbceffff640 1 -- 172.21.15.44:0/4273771629 learned_addr learned my addr 172.21.15.44:0/4273771629 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:48.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.439+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56292956 0 0) 0x7fbcf00b2a30 con 0x7fbcf00a98f0 2024-03-25T17:47:48.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.439+0000 7fbceeffd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:48.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.439+0000 7fbceeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:48.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.439+0000 7fbceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:48.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.439+0000 7fbceeffd640 10 monclient(hunting): my global_id is 4607 2024-03-25T17:47:48.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.439+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd8003520 con 0x7fbcf00a98f0 2024-03-25T17:47:48.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.447+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815947327 0 0) 0x7fbcf0142520 con 0x7fbcf013f710 2024-03-25T17:47:48.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.447+0000 7fbceeffd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:48.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.447+0000 7fbceeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:48.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.447+0000 7fbceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:48.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.447+0000 7fbceeffd640 10 monclient(hunting): my global_id is 4611 2024-03-25T17:47:48.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.447+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf00b2a30 con 0x7fbcf013f710 2024-03-25T17:47:48.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.448+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380006912 0 0) 0x7fbcf00b28a0 con 0x7fbcf01430c0 2024-03-25T17:47:48.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.448+0000 7fbceeffd640 10 monclient(hunting): _negotiate 2024-03-25T17:47:48.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.448+0000 7fbceeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:48.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.448+0000 7fbceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:48.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.448+0000 7fbceeffd640 10 monclient(hunting): my global_id is 4600 2024-03-25T17:47:48.449 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.448+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf0142520 con 0x7fbcf01430c0 2024-03-25T17:47:48.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2904456202 0 0) 0x7fbcd8003520 con 0x7fbcf00a98f0 2024-03-25T17:47:48.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 >> v1:172.21.15.111:6789/0 conn(0x7fbcf01430c0 legacy=0x7fbcf014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 >> v1:172.21.15.44:6789/0 conn(0x7fbcf013f710 legacy=0x7fbcf01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 10 monclient: _finish_hunting 0 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 1 monclient: found mon.noname-b 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf00b44c0 con 0x7fbcf00a98f0 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 10 monclient: _finish_auth 0 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 10 monclient: _check_auth_tickets 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbce0001690 con 0x7fbcf00a98f0 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbceeffd640 10 monclient: dump: 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:48.457 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbcf6732640 5 monclient: authenticate success, global_id 4607 2024-03-25T17:47:48.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.456+0000 7fbcf6732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:48.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.472+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbce00017f0 con 0x7fbcf00a98f0 2024-03-25T17:47:48.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.472+0000 7fbceeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:48.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.472+0000 7fbceeffd640 1 -- 172.21.15.44:0/4273771629 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbce0001c00 con 0x7fbcf00a98f0 2024-03-25T17:47:48.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.472+0000 7fbcf6732640 10 monclient: get_monmap_and_config success 2024-03-25T17:47:48.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.472+0000 7fbcf6732640 4 set_mon_vals no callback set 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.472+0000 7fbceeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.472+0000 7fbceeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.472+0000 7fbceeffd640 10 monclient: dump: 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:48.473 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:48.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 10 monclient: shutdown 2024-03-25T17:47:48.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:48.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 1 -- 172.21.15.44:0/4273771629 >> v1:172.21.15.44:6790/0 conn(0x7fbcf00a98f0 legacy=0x7fbcf00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:48.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 1 -- 172.21.15.44:0/4273771629 shutdown_connections 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 1 -- 172.21.15.44:0/4273771629 >> 172.21.15.44:0/4273771629 conn(0x7fbcf00b7ea0 msgr2=0x7fbcf00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 1 -- 172.21.15.44:0/4273771629 shutdown_connections 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 1 -- 172.21.15.44:0/4273771629 wait complete. 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 10 monclient: build_initial_monmap 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.473+0000 7fbcf6732640 10 monclient: monmap: 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 1 librados: starting msgr at 2024-03-25T17:47:48.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 1 librados: starting objecter 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 1 Processor -- start 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 1 -- start start 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 1 librados: setting wanted keys 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 1 librados: calling monclient init 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 10 monclient: init 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.474+0000 7fbcf6732640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.475+0000 7fbcf6732640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.475+0000 7fbcf6732640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.475+0000 7fbcf6732640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.475+0000 7fbcf6732640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf0137370 con 0x7fbcf013f710 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.475+0000 7fbcf6732640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf01352a0 con 0x7fbcf00a98f0 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.475+0000 7fbcf6732640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf0131760 con 0x7fbcf01430c0 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.475+0000 7fbcf6732640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:48.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.475+0000 7fbcf6732640 10 monclient(hunting): authenticate will time out at 6471.702783s 2024-03-25T17:47:48.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.484+0000 7fbcef7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbcf013f710 0x7fbcf01d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40752/0 (socket says 172.21.15.44:40752) 2024-03-25T17:47:48.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.484+0000 7fbcef7fe640 1 -- 172.21.15.44:0/3117547210 learned_addr learned my addr 172.21.15.44:0/3117547210 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:48.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.487+0000 7fbcecff9640 10 client.?.objecter ms_handle_connect 0x7fbcf013f710 2024-03-25T17:47:48.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.487+0000 7fbcecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:48.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.489+0000 7fbcecff9640 10 client.?.objecter ms_handle_connect 0x7fbcf01430c0 2024-03-25T17:47:48.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.489+0000 7fbcecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:48.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.491+0000 7fbcecff9640 10 client.?.objecter ms_handle_connect 0x7fbcf00a98f0 2024-03-25T17:47:48.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.491+0000 7fbcecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:48.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477207336 0 0) 0x7fbcf01352a0 con 0x7fbcf00a98f0 2024-03-25T17:47:48.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 10 monclient(hunting): _negotiate 2024-03-25T17:47:48.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:48.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:48.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 10 monclient(hunting): my global_id is 4603 2024-03-25T17:47:48.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcdc004eb0 con 0x7fbcf00a98f0 2024-03-25T17:47:48.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029452390 0 0) 0x7fbcf0137370 con 0x7fbcf013f710 2024-03-25T17:47:48.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 10 monclient(hunting): _negotiate 2024-03-25T17:47:48.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:48.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:48.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 10 monclient(hunting): my global_id is 4614 2024-03-25T17:47:48.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.503+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf01352a0 con 0x7fbcf013f710 2024-03-25T17:47:48.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.505+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865932361 0 0) 0x7fbcf0131760 con 0x7fbcf01430c0 2024-03-25T17:47:48.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.505+0000 7fbcecff9640 10 monclient(hunting): _negotiate 2024-03-25T17:47:48.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.505+0000 7fbcecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:48.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.505+0000 7fbcecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:48.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.505+0000 7fbcecff9640 10 monclient(hunting): my global_id is 4610 2024-03-25T17:47:48.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.505+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf0137370 con 0x7fbcf01430c0 2024-03-25T17:47:48.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.515+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 106740852 0 0) 0x7fbcf0137370 con 0x7fbcf01430c0 2024-03-25T17:47:48.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.515+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcf0131760 con 0x7fbcf01430c0 2024-03-25T17:47:48.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.516+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbce8001bd0 con 0x7fbcf01430c0 2024-03-25T17:47:48.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.516+0000 7fbcecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:48.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.516+0000 7fbcecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:47:48.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.516+0000 7fbcecff9640 10 monclient(hunting): dump: 2024-03-25T17:47:48.516 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:48.516 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:48.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.516+0000 7fbcecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.520+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2163725331 0 0) 0x7fbcdc004eb0 con 0x7fbcf00a98f0 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.520+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcf0137370 con 0x7fbcf00a98f0 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.520+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbce0001b00 con 0x7fbcf00a98f0 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.520+0000 7fbcecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.520+0000 7fbcecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.520+0000 7fbcecff9640 10 monclient(hunting): dump: 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:48.521 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:48.522 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:48.522 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:48.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 365616678 0 0) 0x7fbcf0131760 con 0x7fbcf01430c0 2024-03-25T17:47:48.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 >> v1:172.21.15.111:6789/0 conn(0x7fbcf00a98f0 legacy=0x7fbcf0131050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:48.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 >> v1:172.21.15.44:6789/0 conn(0x7fbcf013f710 legacy=0x7fbcf01d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:48.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 10 monclient: _finish_hunting 0 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 1 monclient: found mon.c 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf01318f0 con 0x7fbcf01430c0 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 10 monclient: _finish_auth 0 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 10 monclient: _check_auth_tickets 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcf6732640 5 monclient: authenticate success, global_id 4610 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcf6732640 10 monclient: _renew_subs 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcf6732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.529+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf00b7a40 con 0x7fbcf01430c0 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.530+0000 7fbcf6732640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.530+0000 7fbcf6732640 10 monclient: _renew_subs 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.530+0000 7fbcf6732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:48.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.530+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf01360e0 con 0x7fbcf01430c0 2024-03-25T17:47:48.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.530+0000 7fbcf6732640 1 librados: init done 2024-03-25T17:47:48.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.531+0000 7fbcf6732640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:48.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.531+0000 7fbcf6732640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:48.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.531+0000 7fbcf6732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:48.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.531+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf0135430 con 0x7fbcf01430c0 2024-03-25T17:47:48.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.746+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbce8001fe0 con 0x7fbcf01430c0 2024-03-25T17:47:48.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.746+0000 7fbcecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:48.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.746+0000 7fbcee7fc640 4 set_mon_vals no callback set 2024-03-25T17:47:48.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.838+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbce8003b30 con 0x7fbcf01430c0 2024-03-25T17:47:48.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.838+0000 7fbcecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.838+0000 7fbcecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.838+0000 7fbcecff9640 10 monclient: dump: 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:48.839 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:48.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.839+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fbce8019b00 con 0x7fbcf01430c0 2024-03-25T17:47:48.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.839+0000 7fbceffff640 10 monclient: get_auth_request con 0x7fbcdc06df10 auth_method 0 2024-03-25T17:47:48.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.840+0000 7fbcecff9640 10 client.4610.objecter ms_handle_connect 0x7fbcdc06df10 2024-03-25T17:47:48.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.848+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fbce80537a0 con 0x7fbcf01430c0 2024-03-25T17:47:48.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.848+0000 7fbcecff9640 10 client.4610.objecter ms_dispatch 0x7fbcf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:47:48.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.848+0000 7fbcecff9640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:47:48.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.848+0000 7fbcecff9640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:47:48.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.848+0000 7fbcecff9640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-25T17:47:48.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.848+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbce804f2b0 con 0x7fbcf01430c0 2024-03-25T17:47:48.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.848+0000 7fbcecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:48.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:48.848+0000 7fbcecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:47:49.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.109+0000 7fbcf6732640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:49.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.109+0000 7fbcf6732640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:49.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.109+0000 7fbcf6732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:47:49.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.109+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcf00627b0 con 0x7fbcf01430c0 2024-03-25T17:47:49.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.236+0000 7fbcecff9640 1 -- 172.21.15.44:0/3117547210 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fbce80511e0 con 0x7fbcf01430c0 2024-03-25T17:47:49.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.236+0000 7fbcecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:49.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.236+0000 7fbcecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:47:49.237 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:47:49.237 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.238+0000 7fbcf6732640 10 librados: watch_flush enter 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 10 librados: watch_flush exit 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 10 client.4610.objecter successfully canceled tick 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 >> v1:172.21.15.44:6800/3257043132 conn(0x7fbcdc06df10 legacy=0x7fbcdc0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 10 monclient: shutdown 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 >> v1:172.21.15.44:6790/0 conn(0x7fbcf01430c0 legacy=0x7fbcf01d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 shutdown_connections 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 >> 172.21.15.44:0/3117547210 conn(0x7fbcf00b7ea0 msgr2=0x7fbcf0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 shutdown_connections 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 1 -- 172.21.15.44:0/3117547210 wait complete. 2024-03-25T17:47:49.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:49.239+0000 7fbcf6732640 1 librados: shutdown 2024-03-25T17:47:49.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:47:56.057 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:47:56.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.169+0000 7f73ed40f640 10 monclient: get_monmap_and_config 2024-03-25T17:47:56.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.169+0000 7f73ed40f640 10 monclient: build_initial_monmap 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.169+0000 7f73ed40f640 10 monclient: monmap: 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:47:56.171239+0000 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:47:56.171239+0000 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:47:56.171 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:56.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.169+0000 7f73ed40f640 1 Processor -- start 2024-03-25T17:47:56.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.169+0000 7f73ed40f640 1 -- start start 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.169+0000 7f73ed40f640 10 monclient: init 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 10 monclient(hunting): picked mon.noname-c con 0x7f73e80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 10 monclient(hunting): picked mon.noname-a con 0x7f73e813f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 10 monclient(hunting): picked mon.noname-b con 0x7f73e81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e8142520 con 0x7f73e813f710 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e80b28a0 con 0x7f73e80a98f0 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e80b2a30 con 0x7f73e81430c0 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.170+0000 7f73ed40f640 10 monclient(hunting): authenticate will time out at 6479.399249s 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.171+0000 7f73e6ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f73e80a98f0 0x7f73e80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:51540/0 (socket says 172.21.15.44:51540) 2024-03-25T17:47:56.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.171+0000 7f73e6ffd640 1 -- 172.21.15.44:0/2515034888 learned_addr learned my addr 172.21.15.44:0/2515034888 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:56.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.251+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238364776 0 0) 0x7f73e8142520 con 0x7f73e813f710 2024-03-25T17:47:56.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.251+0000 7f73e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:47:56.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.251+0000 7f73e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:56.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.251+0000 7f73e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:56.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.251+0000 7f73e5ffb640 10 monclient(hunting): my global_id is 4617 2024-03-25T17:47:56.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.251+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d0003520 con 0x7f73e813f710 2024-03-25T17:47:56.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.266+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1085305599 0 0) 0x7f73d0003520 con 0x7f73e813f710 2024-03-25T17:47:56.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 >> v1:172.21.15.44:6790/0 conn(0x7f73e81430c0 legacy=0x7f73e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 >> v1:172.21.15.111:6789/0 conn(0x7f73e80a98f0 legacy=0x7f73e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 1 monclient: found mon.noname-a 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e80b44c0 con 0x7f73e813f710 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 10 monclient: _finish_auth 0 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f73dc001690 con 0x7f73e813f710 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73ed40f640 5 monclient: authenticate success, global_id 4617 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73ed40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:47:56.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73e5ffb640 10 monclient: dump: 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:56.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.267+0000 7f73ed40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:47:56.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.272+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73dc0017f0 con 0x7f73e813f710 2024-03-25T17:47:56.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.272+0000 7f73e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:56.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.272+0000 7f73e5ffb640 1 -- 172.21.15.44:0/2515034888 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f73dc001c00 con 0x7f73e813f710 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.272+0000 7f73ed40f640 10 monclient: get_monmap_and_config success 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.272+0000 7f73e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.272+0000 7f73ed40f640 4 set_mon_vals no callback set 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.272+0000 7f73e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.272+0000 7f73e5ffb640 10 monclient: dump: 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:56.274 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:56.275 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:56.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 10 monclient: shutdown 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 1 -- 172.21.15.44:0/2515034888 >> v1:172.21.15.44:6789/0 conn(0x7f73e813f710 legacy=0x7f73e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 1 -- 172.21.15.44:0/2515034888 shutdown_connections 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 1 -- 172.21.15.44:0/2515034888 >> 172.21.15.44:0/2515034888 conn(0x7f73e80b7ea0 msgr2=0x7f73e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 1 -- 172.21.15.44:0/2515034888 shutdown_connections 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 1 -- 172.21.15.44:0/2515034888 wait complete. 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 10 monclient: build_initial_monmap 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.273+0000 7f73ed40f640 10 monclient: monmap: 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:47:56.277 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.274+0000 7f73ed40f640 1 librados: starting msgr at 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.274+0000 7f73ed40f640 1 librados: starting objecter 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.274+0000 7f73ed40f640 1 Processor -- start 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.274+0000 7f73ed40f640 1 -- start start 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.274+0000 7f73ed40f640 1 librados: setting wanted keys 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.274+0000 7f73ed40f640 1 librados: calling monclient init 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.274+0000 7f73ed40f640 10 monclient: init 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 10 monclient: _reopen_session rank -1 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 10 monclient(hunting): picked mon.noname-a con 0x7f73e80a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 10 monclient(hunting): picked mon.noname-b con 0x7f73e813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:47:56.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 10 monclient(hunting): picked mon.noname-c con 0x7f73e81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:47:56.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e8139a80 con 0x7f73e80a98f0 2024-03-25T17:47:56.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e8137450 con 0x7f73e813f710 2024-03-25T17:47:56.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e81d7ba0 con 0x7f73e81430c0 2024-03-25T17:47:56.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 10 monclient(hunting): _renew_subs 2024-03-25T17:47:56.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.275+0000 7f73ed40f640 10 monclient(hunting): authenticate will time out at 6479.503807s 2024-03-25T17:47:56.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.283+0000 7f73e6ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f73e80a98f0 0x7f73e8139370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55568/0 (socket says 172.21.15.44:55568) 2024-03-25T17:47:56.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.284+0000 7f73e6ffd640 1 -- 172.21.15.44:0/3842501420 learned_addr learned my addr 172.21.15.44:0/3842501420 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:47:56.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.285+0000 7f73cf7fe640 10 client.?.objecter ms_handle_connect 0x7f73e80a98f0 2024-03-25T17:47:56.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.285+0000 7f73cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:56.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.296+0000 7f73cf7fe640 10 client.?.objecter ms_handle_connect 0x7f73e813f710 2024-03-25T17:47:56.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.296+0000 7f73cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:56.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.310+0000 7f73cf7fe640 10 client.?.objecter ms_handle_connect 0x7f73e81430c0 2024-03-25T17:47:56.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.310+0000 7f73cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:47:56.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.312+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486438501 0 0) 0x7f73e80b28a0 con 0x7f73e80a98f0 2024-03-25T17:47:56.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.312+0000 7f73cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:47:56.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.312+0000 7f73cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:56.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.312+0000 7f73cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:56.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.312+0000 7f73cf7fe640 10 monclient(hunting): my global_id is 4620 2024-03-25T17:47:56.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.312+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d4004eb0 con 0x7f73e80a98f0 2024-03-25T17:47:56.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.324+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725905623 0 0) 0x7f73e8137450 con 0x7f73e813f710 2024-03-25T17:47:56.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.324+0000 7f73cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:47:56.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.324+0000 7f73cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:56.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.324+0000 7f73cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:56.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.324+0000 7f73cf7fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T17:47:56.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.324+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e80b28a0 con 0x7f73e813f710 2024-03-25T17:47:56.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.329+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437350501 0 0) 0x7f73e81d7ba0 con 0x7f73e81430c0 2024-03-25T17:47:56.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.329+0000 7f73cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:47:56.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.329+0000 7f73cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:47:56.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.329+0000 7f73cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:47:56.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.329+0000 7f73cf7fe640 10 monclient(hunting): my global_id is 4616 2024-03-25T17:47:56.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.329+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e8137450 con 0x7f73e81430c0 2024-03-25T17:47:56.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.339+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518916134 0 0) 0x7f73e80b28a0 con 0x7f73e813f710 2024-03-25T17:47:56.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.339+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73e81d7ba0 con 0x7f73e813f710 2024-03-25T17:47:56.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.339+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f73dc001b80 con 0x7f73e813f710 2024-03-25T17:47:56.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.339+0000 7f73cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:56.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.339+0000 7f73cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:47:56.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.339+0000 7f73cf7fe640 10 monclient(hunting): dump: 2024-03-25T17:47:56.340 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:56.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.339+0000 7f73cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:47:56.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.343+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3178332413 0 0) 0x7f73d4004eb0 con 0x7f73e80a98f0 2024-03-25T17:47:56.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.343+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73e80b28a0 con 0x7f73e80a98f0 2024-03-25T17:47:56.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.343+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f73d8001710 con 0x7f73e80a98f0 2024-03-25T17:47:56.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.343+0000 7f73cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:56.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.343+0000 7f73cf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.343+0000 7f73cf7fe640 10 monclient(hunting): dump: 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:56.345 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:56.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3984596437 0 0) 0x7f73e81d7ba0 con 0x7f73e813f710 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 >> v1:172.21.15.44:6790/0 conn(0x7f73e81430c0 legacy=0x7f73e8132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 >> v1:172.21.15.44:6789/0 conn(0x7f73e80a98f0 legacy=0x7f73e8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 1 monclient: found mon.b 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e81368f0 con 0x7f73e813f710 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 10 monclient: _finish_auth 0 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 5 monclient: authenticate success, global_id 4609 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 10 monclient: _renew_subs 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:56.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e8136b40 con 0x7f73e813f710 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 10 monclient: _renew_subs 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e8137c00 con 0x7f73e813f710 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.344+0000 7f73ed40f640 1 librados: init done 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.345+0000 7f73ed40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.345+0000 7f73ed40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.345+0000 7f73ed40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:56.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.345+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e80627b0 con 0x7f73e813f710 2024-03-25T17:47:56.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.562+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73dc0038a0 con 0x7f73e813f710 2024-03-25T17:47:56.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.562+0000 7f73cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:47:56.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.562+0000 7f73e57fa640 4 set_mon_vals no callback set 2024-03-25T17:47:56.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.616+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f73dc003a10 con 0x7f73e813f710 2024-03-25T17:47:56.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.617+0000 7f73cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.617+0000 7f73cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.617+0000 7f73cf7fe640 10 monclient: dump: 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:47:56.619 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:47:56.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.617+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f73dc0199e0 con 0x7f73e813f710 2024-03-25T17:47:56.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.618+0000 7f73e6ffd640 10 monclient: get_auth_request con 0x7f73d406df70 auth_method 0 2024-03-25T17:47:56.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.618+0000 7f73cf7fe640 10 client.4609.objecter ms_handle_connect 0x7f73d406df70 2024-03-25T17:47:56.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.620+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f73dc053610 con 0x7f73e813f710 2024-03-25T17:47:56.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.620+0000 7f73cf7fe640 10 client.4609.objecter ms_dispatch 0x7f73e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:47:56.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.620+0000 7f73cf7fe640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:47:56.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.620+0000 7f73cf7fe640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:47:56.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.620+0000 7f73cf7fe640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T17:47:56.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.622+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73dc04ef50 con 0x7f73e813f710 2024-03-25T17:47:56.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.622+0000 7f73cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:47:56.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.622+0000 7f73cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:47:56.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.886+0000 7f73ed40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:56.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.886+0000 7f73ed40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:56.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.886+0000 7f73ed40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:47:56.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.886+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73e8074cc0 con 0x7f73e813f710 2024-03-25T17:47:56.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.909+0000 7f73cf7fe640 1 -- 172.21.15.44:0/3842501420 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f73dc051050 con 0x7f73e813f710 2024-03-25T17:47:56.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.909+0000 7f73cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:47:56.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.909+0000 7f73cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:47:56.911 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:47:56.911 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 10 librados: watch_flush enter 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 10 librados: watch_flush exit 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 10 client.4609.objecter successfully canceled tick 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 >> v1:172.21.15.44:6800/3257043132 conn(0x7f73d406df70 legacy=0x7f73d4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 10 monclient: shutdown 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 >> v1:172.21.15.111:6789/0 conn(0x7f73e813f710 legacy=0x7f73e81d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 shutdown_connections 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 >> 172.21.15.44:0/3842501420 conn(0x7f73e80b7ea0 msgr2=0x7f73e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 shutdown_connections 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.912+0000 7f73ed40f640 1 -- 172.21.15.44:0/3842501420 wait complete. 2024-03-25T17:47:56.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:47:56.913+0000 7f73ed40f640 1 librados: shutdown 2024-03-25T17:47:56.927 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:47:58.915 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:16.271 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:58.972 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:16.328 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:59.005 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:16.361 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:59.013 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:16.333 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:59.021 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:16.341 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:59.039 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:16.394 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:59.046 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:16.402 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:59.053 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:16.373 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:47:59.078 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:16.433 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:03.733 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.847+0000 7f1acc44e640 10 monclient: get_monmap_and_config 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.848+0000 7f1acc44e640 10 monclient: build_initial_monmap 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.848+0000 7f1acc44e640 10 monclient: monmap: 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:48:03.848943+0000 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:48:03.848943+0000 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:03.849 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:03.850 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:03.850 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:48:03.850 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:48:03.850 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:03.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.848+0000 7f1acc44e640 1 Processor -- start 2024-03-25T17:48:03.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.848+0000 7f1acc44e640 1 -- start start 2024-03-25T17:48:03.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.848+0000 7f1acc44e640 10 monclient: init 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 10 monclient(hunting): picked mon.noname-c con 0x7f1ac40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 10 monclient(hunting): picked mon.noname-b con 0x7f1ac40b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 10 monclient(hunting): picked mon.noname-a con 0x7f1ac414be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40af940 con 0x7f1ac414be90 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40b0a70 con 0x7f1ac40b4d60 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40b0d50 con 0x7f1ac40b1730 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:03.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.849+0000 7f1acc44e640 10 monclient(hunting): authenticate will time out at 6487.077427s 2024-03-25T17:48:03.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.857+0000 7f1ac99e9640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f1ac40b1730 0x7f1ac40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:45270/0 (socket says 172.21.15.44:45270) 2024-03-25T17:48:03.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.857+0000 7f1ac99e9640 1 -- 172.21.15.44:0/3727549540 learned_addr learned my addr 172.21.15.44:0/3727549540 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:03.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.878+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011214312 0 0) 0x7f1ac40b0d50 con 0x7f1ac40b1730 2024-03-25T17:48:03.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.878+0000 7f1ac91e8640 10 monclient(hunting): _negotiate 2024-03-25T17:48:03.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.878+0000 7f1ac91e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:03.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.878+0000 7f1ac91e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:03.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.878+0000 7f1ac91e8640 10 monclient(hunting): my global_id is 4619 2024-03-25T17:48:03.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.878+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab0003520 con 0x7f1ac40b1730 2024-03-25T17:48:03.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.882+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024772765 0 0) 0x7f1ac40b0a70 con 0x7f1ac40b4d60 2024-03-25T17:48:03.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.882+0000 7f1ac91e8640 10 monclient(hunting): _negotiate 2024-03-25T17:48:03.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.882+0000 7f1ac91e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:03.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.882+0000 7f1ac91e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:03.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.882+0000 7f1ac91e8640 10 monclient(hunting): my global_id is 4612 2024-03-25T17:48:03.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.882+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac40b0d50 con 0x7f1ac40b4d60 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.886+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2584804739 0 0) 0x7f1ab0003520 con 0x7f1ac40b1730 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 >> v1:172.21.15.111:6789/0 conn(0x7f1ac40b4d60 legacy=0x7f1ac413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 >> v1:172.21.15.44:6789/0 conn(0x7f1ac414be90 legacy=0x7f1ac414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 10 monclient: _finish_hunting 0 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 1 monclient: found mon.noname-b 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40b1030 con 0x7f1ac40b1730 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 10 monclient: _finish_auth 0 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 10 monclient: _check_auth_tickets 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1ab4001690 con 0x7f1ac40b1730 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1acc44e640 5 monclient: authenticate success, global_id 4619 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1acc44e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:03.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1ac91e8640 10 monclient: dump: 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:03.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.887+0000 7f1acc44e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:03.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.922+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ab40017f0 con 0x7f1ac40b1730 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.922+0000 7f1ac91e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.922+0000 7f1ac91e8640 1 -- 172.21.15.44:0/3727549540 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1ab4001c00 con 0x7f1ac40b1730 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.922+0000 7f1acc44e640 10 monclient: get_monmap_and_config success 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.922+0000 7f1acc44e640 4 set_mon_vals no callback set 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.922+0000 7f1ac91e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.922+0000 7f1ac91e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.922+0000 7f1ac91e8640 10 monclient: dump: 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:03.923 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:03.924 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:03.924 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:03.924 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.923+0000 7f1acc44e640 10 monclient: shutdown 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.923+0000 7f1acc44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.923+0000 7f1acc44e640 1 -- 172.21.15.44:0/3727549540 >> v1:172.21.15.44:6790/0 conn(0x7f1ac40b1730 legacy=0x7f1ac40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.923+0000 7f1acc44e640 1 -- 172.21.15.44:0/3727549540 shutdown_connections 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.923+0000 7f1acc44e640 1 -- 172.21.15.44:0/3727549540 >> 172.21.15.44:0/3727549540 conn(0x7f1ac40617a0 msgr2=0x7f1ac4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.923+0000 7f1acc44e640 1 -- 172.21.15.44:0/3727549540 shutdown_connections 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.923+0000 7f1acc44e640 1 -- 172.21.15.44:0/3727549540 wait complete. 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 10 monclient: build_initial_monmap 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 10 monclient: monmap: 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:03.925 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 1 librados: starting msgr at 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 1 librados: starting objecter 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 1 Processor -- start 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 1 -- start start 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 1 librados: setting wanted keys 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 1 librados: calling monclient init 2024-03-25T17:48:03.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.924+0000 7f1acc44e640 10 monclient: init 2024-03-25T17:48:03.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:03.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 10 monclient(hunting): picked mon.noname-b con 0x7f1ac40b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 10 monclient(hunting): picked mon.noname-c con 0x7f1ac40b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 10 monclient(hunting): picked mon.noname-a con 0x7f1ac414be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac4136250 con 0x7f1ac414be90 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac4136af0 con 0x7f1ac40b1730 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac4136c80 con 0x7f1ac40b4d60 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.925+0000 7f1acc44e640 10 monclient(hunting): authenticate will time out at 6487.153344s 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.928+0000 7f1aca9eb640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1ac414be90 0x7f1ac41e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51454/0 (socket says 172.21.15.44:51454) 2024-03-25T17:48:03.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.928+0000 7f1aca9eb640 1 -- 172.21.15.44:0/171718693 learned_addr learned my addr 172.21.15.44:0/171718693 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:03.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.931+0000 7f1abaffd640 10 client.?.objecter ms_handle_connect 0x7f1ac414be90 2024-03-25T17:48:03.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.931+0000 7f1abaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:03.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.945+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413515646 0 0) 0x7f1ac4136250 con 0x7f1ac414be90 2024-03-25T17:48:03.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.945+0000 7f1abaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:03.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.945+0000 7f1abaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:03.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.945+0000 7f1abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:03.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.945+0000 7f1abaffd640 10 monclient(hunting): my global_id is 4626 2024-03-25T17:48:03.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.945+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa8004eb0 con 0x7f1ac414be90 2024-03-25T17:48:03.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.945+0000 7f1abaffd640 10 client.?.objecter ms_handle_connect 0x7f1ac40b4d60 2024-03-25T17:48:03.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.945+0000 7f1abaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:03.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.948+0000 7f1abaffd640 10 client.?.objecter ms_handle_connect 0x7f1ac40b1730 2024-03-25T17:48:03.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.948+0000 7f1abaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:03.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.957+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709779269 0 0) 0x7f1ac4136c80 con 0x7f1ac40b4d60 2024-03-25T17:48:03.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.957+0000 7f1abaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:03.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.957+0000 7f1abaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:03.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.957+0000 7f1abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:03.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.957+0000 7f1abaffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T17:48:03.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.957+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac4136250 con 0x7f1ac40b4d60 2024-03-25T17:48:03.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458034748 0 0) 0x7f1ac4136af0 con 0x7f1ac40b1730 2024-03-25T17:48:03.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:03.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:03.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:03.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 10 monclient(hunting): my global_id is 4615 2024-03-25T17:48:03.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac4136c80 con 0x7f1ac40b1730 2024-03-25T17:48:03.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2857603777 0 0) 0x7f1aa8004eb0 con 0x7f1ac414be90 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ac4136af0 con 0x7f1ac414be90 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1abc001c30 con 0x7f1ac414be90 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 10 monclient(hunting): dump: 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:03.960 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.958+0000 7f1abaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:48:03.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.982+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3473003013 0 0) 0x7f1ac4136c80 con 0x7f1ac40b1730 2024-03-25T17:48:03.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.982+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1aa8004eb0 con 0x7f1ac40b1730 2024-03-25T17:48:03.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.982+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1ac0001930 con 0x7f1ac40b1730 2024-03-25T17:48:03.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.982+0000 7f1abaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.982+0000 7f1abaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.982+0000 7f1abaffd640 10 monclient(hunting): dump: 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:03.984 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 578852791 0 0) 0x7f1ac4136af0 con 0x7f1ac414be90 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 >> v1:172.21.15.44:6790/0 conn(0x7f1ac40b4d60 legacy=0x7f1ac412fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 >> v1:172.21.15.111:6789/0 conn(0x7f1ac40b1730 legacy=0x7f1ac41d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 10 monclient: _finish_hunting 0 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 1 monclient: found mon.a 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac4138210 con 0x7f1ac414be90 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.986+0000 7f1abaffd640 10 monclient: _finish_auth 0 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1abaffd640 10 monclient: _check_auth_tickets 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1abaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 5 monclient: authenticate success, global_id 4626 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 10 monclient: _renew_subs 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac41384c0 con 0x7f1ac414be90 2024-03-25T17:48:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:48:03.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 10 monclient: _renew_subs 2024-03-25T17:48:03.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:03.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac4145700 con 0x7f1ac414be90 2024-03-25T17:48:03.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.987+0000 7f1acc44e640 1 librados: init done 2024-03-25T17:48:03.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.988+0000 7f1acc44e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:03.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.988+0000 7f1acc44e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:03.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.988+0000 7f1acc44e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:03.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:03.988+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac41360a0 con 0x7f1ac414be90 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.299+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1abc003990 con 0x7f1ac414be90 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.299+0000 7f1abaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.299+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f1abc003da0 con 0x7f1ac414be90 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.299+0000 7f1abaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.299+0000 7f1abaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.299+0000 7f1abaffd640 10 monclient: dump: 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:04.300 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:04.301 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:04.301 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:04.301 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:04.301 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:04.301 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:04.301 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:04.301 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:04.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.299+0000 7f1ac89e7640 4 set_mon_vals no callback set 2024-03-25T17:48:04.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.300+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f1abc003fc0 con 0x7f1ac414be90 2024-03-25T17:48:04.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.300+0000 7f1aca1ea640 10 monclient: get_auth_request con 0x7f1aa806df70 auth_method 0 2024-03-25T17:48:04.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.301+0000 7f1abaffd640 10 client.4626.objecter ms_handle_connect 0x7f1aa806df70 2024-03-25T17:48:04.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.307+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f1abc0549c0 con 0x7f1ac414be90 2024-03-25T17:48:04.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.307+0000 7f1abaffd640 10 client.4626.objecter ms_dispatch 0x7f1ac4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:48:04.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.307+0000 7f1abaffd640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:48:04.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.307+0000 7f1abaffd640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:48:04.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.307+0000 7f1abaffd640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T17:48:04.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.307+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1abc0504d0 con 0x7f1ac414be90 2024-03-25T17:48:04.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.307+0000 7f1abaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:04.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.307+0000 7f1abaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:48:04.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.525+0000 7f1acc44e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:04.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.525+0000 7f1acc44e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:04.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.525+0000 7f1acc44e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:04.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.525+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ac4135ec0 con 0x7f1ac414be90 2024-03-25T17:48:04.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.612+0000 7f1abaffd640 1 -- 172.21.15.44:0/171718693 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f1abc052400 con 0x7f1ac414be90 2024-03-25T17:48:04.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.612+0000 7f1abaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:04.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.612+0000 7f1abaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:48:04.613 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:48:04.613 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.614+0000 7f1acc44e640 10 librados: watch_flush enter 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 10 librados: watch_flush exit 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 10 client.4626.objecter successfully canceled tick 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 >> v1:172.21.15.44:6800/3257043132 conn(0x7f1aa806df70 legacy=0x7f1aa8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 10 monclient: shutdown 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 >> v1:172.21.15.44:6789/0 conn(0x7f1ac414be90 legacy=0x7f1ac41e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 shutdown_connections 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 >> 172.21.15.44:0/171718693 conn(0x7f1ac40617a0 msgr2=0x7f1ac40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 shutdown_connections 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 1 -- 172.21.15.44:0/171718693 wait complete. 2024-03-25T17:48:04.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:04.615+0000 7f1acc44e640 1 librados: shutdown 2024-03-25T17:48:04.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:48:11.434 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:48:11.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.550+0000 7fdb0e70f640 10 monclient: get_monmap_and_config 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.550+0000 7fdb0e70f640 10 monclient: build_initial_monmap 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.550+0000 7fdb0e70f640 10 monclient: monmap: 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:48:11.551235+0000 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:48:11.551235+0000 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:48:11.551 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.551+0000 7fdb0e70f640 1 Processor -- start 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.551+0000 7fdb0e70f640 1 -- start start 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.551+0000 7fdb0e70f640 10 monclient: init 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.551+0000 7fdb0e70f640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.551+0000 7fdb0e70f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.551+0000 7fdb0e70f640 10 monclient(hunting): picked mon.noname-a con 0x7fdb080a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.551+0000 7fdb0e70f640 10 monclient(hunting): picked mon.noname-c con 0x7fdb0813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.552+0000 7fdb0e70f640 10 monclient(hunting): picked mon.noname-b con 0x7fdb081430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.552+0000 7fdb0e70f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb08142520 con 0x7fdb080a98f0 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.552+0000 7fdb0e70f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb080b28a0 con 0x7fdb0813f710 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.552+0000 7fdb0e70f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb080b2a30 con 0x7fdb081430c0 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.552+0000 7fdb0e70f640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:11.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.552+0000 7fdb0e70f640 10 monclient(hunting): authenticate will time out at 6494.779804s 2024-03-25T17:48:11.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.571+0000 7fdb077fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdb0813f710 0x7fdb081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:33624/0 (socket says 172.21.15.44:33624) 2024-03-25T17:48:11.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.571+0000 7fdb077fe640 1 -- 172.21.15.44:0/48341234 learned_addr learned my addr 172.21.15.44:0/48341234 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:11.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.584+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505486207 0 0) 0x7fdb080b28a0 con 0x7fdb0813f710 2024-03-25T17:48:11.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.584+0000 7fdb06ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:11.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.584+0000 7fdb06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:11.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.584+0000 7fdb06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:11.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.584+0000 7fdb06ffd640 10 monclient(hunting): my global_id is 4618 2024-03-25T17:48:11.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.584+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf0003520 con 0x7fdb0813f710 2024-03-25T17:48:11.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.595+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180804862 0 0) 0x7fdb08142520 con 0x7fdb080a98f0 2024-03-25T17:48:11.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.595+0000 7fdb06ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:11.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.595+0000 7fdb06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:11.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.595+0000 7fdb06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:11.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.595+0000 7fdb06ffd640 10 monclient(hunting): my global_id is 4629 2024-03-25T17:48:11.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.595+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb080b28a0 con 0x7fdb080a98f0 2024-03-25T17:48:11.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.595+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461281268 0 0) 0x7fdb080b2a30 con 0x7fdb081430c0 2024-03-25T17:48:11.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.596+0000 7fdb06ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:11.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.596+0000 7fdb06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:11.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.596+0000 7fdb06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:11.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.596+0000 7fdb06ffd640 10 monclient(hunting): my global_id is 4625 2024-03-25T17:48:11.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.596+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb08142520 con 0x7fdb081430c0 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2933127166 0 0) 0x7fdb080b28a0 con 0x7fdb080a98f0 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 >> v1:172.21.15.44:6790/0 conn(0x7fdb081430c0 legacy=0x7fdb0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 >> v1:172.21.15.111:6789/0 conn(0x7fdb0813f710 legacy=0x7fdb081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 1 monclient: found mon.noname-a 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb080b44c0 con 0x7fdb080a98f0 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 10 monclient: _finish_auth 0 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:48:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdaf8001670 con 0x7fdb080a98f0 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb06ffd640 10 monclient: dump: 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb0e70f640 5 monclient: authenticate success, global_id 4629 2024-03-25T17:48:11.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.605+0000 7fdb0e70f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.626+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdaf80017d0 con 0x7fdb080a98f0 2024-03-25T17:48:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.626+0000 7fdb06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.626+0000 7fdb06ffd640 1 -- 172.21.15.44:0/48341234 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdaf8001be0 con 0x7fdb080a98f0 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.626+0000 7fdb0e70f640 10 monclient: get_monmap_and_config success 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.626+0000 7fdb0e70f640 4 set_mon_vals no callback set 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.626+0000 7fdb06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.626+0000 7fdb06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.626+0000 7fdb06ffd640 10 monclient: dump: 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:11.627 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.627+0000 7fdb0e70f640 10 monclient: shutdown 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.627+0000 7fdb0e70f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.627+0000 7fdb0e70f640 1 -- 172.21.15.44:0/48341234 >> v1:172.21.15.44:6789/0 conn(0x7fdb080a98f0 legacy=0x7fdb080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.627+0000 7fdb0e70f640 1 -- 172.21.15.44:0/48341234 shutdown_connections 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.627+0000 7fdb0e70f640 1 -- 172.21.15.44:0/48341234 >> 172.21.15.44:0/48341234 conn(0x7fdb080b7ea0 msgr2=0x7fdb080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.627+0000 7fdb0e70f640 1 -- 172.21.15.44:0/48341234 shutdown_connections 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.627+0000 7fdb0e70f640 1 -- 172.21.15.44:0/48341234 wait complete. 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 10 monclient: build_initial_monmap 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 10 monclient: monmap: 2024-03-25T17:48:11.629 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 1 librados: starting msgr at 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 1 librados: starting objecter 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 1 Processor -- start 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 1 -- start start 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 1 librados: setting wanted keys 2024-03-25T17:48:11.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 1 librados: calling monclient init 2024-03-25T17:48:11.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.628+0000 7fdb0e70f640 10 monclient: init 2024-03-25T17:48:11.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:11.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:48:11.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 10 monclient(hunting): picked mon.noname-a con 0x7fdb080a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:11.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 10 monclient(hunting): picked mon.noname-c con 0x7fdb0813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:11.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 10 monclient(hunting): picked mon.noname-b con 0x7fdb081430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:11.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb08137030 con 0x7fdb080a98f0 2024-03-25T17:48:11.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb081398a0 con 0x7fdb081430c0 2024-03-25T17:48:11.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb081d4c90 con 0x7fdb0813f710 2024-03-25T17:48:11.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:11.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.629+0000 7fdb0e70f640 10 monclient(hunting): authenticate will time out at 6494.856969s 2024-03-25T17:48:11.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.650+0000 7fdb0ccac640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdb081430c0 0x7fdb081d7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:33676/0 (socket says 172.21.15.44:33676) 2024-03-25T17:48:11.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.650+0000 7fdb0ccac640 1 -- 172.21.15.44:0/2487309123 learned_addr learned my addr 172.21.15.44:0/2487309123 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:11.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.651+0000 7fdb04ff9640 10 client.?.objecter ms_handle_connect 0x7fdb081430c0 2024-03-25T17:48:11.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.651+0000 7fdb04ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:11.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.659+0000 7fdb04ff9640 10 client.?.objecter ms_handle_connect 0x7fdb080a98f0 2024-03-25T17:48:11.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.659+0000 7fdb04ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:11.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.659+0000 7fdb04ff9640 10 client.?.objecter ms_handle_connect 0x7fdb0813f710 2024-03-25T17:48:11.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.659+0000 7fdb04ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:11.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.660+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644293785 0 0) 0x7fdb081398a0 con 0x7fdb081430c0 2024-03-25T17:48:11.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.660+0000 7fdb04ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:48:11.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.660+0000 7fdb04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:11.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.660+0000 7fdb04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:11.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.660+0000 7fdb04ff9640 10 monclient(hunting): my global_id is 4621 2024-03-25T17:48:11.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.660+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf4004eb0 con 0x7fdb081430c0 2024-03-25T17:48:11.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.683+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678053442 0 0) 0x7fdb081d4c90 con 0x7fdb0813f710 2024-03-25T17:48:11.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.683+0000 7fdb04ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:48:11.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.683+0000 7fdb04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:11.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.683+0000 7fdb04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:11.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.683+0000 7fdb04ff9640 10 monclient(hunting): my global_id is 4628 2024-03-25T17:48:11.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.683+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb081398a0 con 0x7fdb0813f710 2024-03-25T17:48:11.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.687+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3135938563 0 0) 0x7fdaf4004eb0 con 0x7fdb081430c0 2024-03-25T17:48:11.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.687+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb081d4c90 con 0x7fdb081430c0 2024-03-25T17:48:11.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.687+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdb00001c70 con 0x7fdb081430c0 2024-03-25T17:48:11.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:11.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:48:11.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 10 monclient(hunting): dump: 2024-03-25T17:48:11.688 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:11.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:48:11.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347313429 0 0) 0x7fdb08137030 con 0x7fdb080a98f0 2024-03-25T17:48:11.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:48:11.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:11.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:11.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 10 monclient(hunting): my global_id is 4632 2024-03-25T17:48:11.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.688+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf4004eb0 con 0x7fdb080a98f0 2024-03-25T17:48:11.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.700+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378869042 0 0) 0x7fdb081398a0 con 0x7fdb0813f710 2024-03-25T17:48:11.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.700+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb08137030 con 0x7fdb0813f710 2024-03-25T17:48:11.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.700+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdafc001950 con 0x7fdb0813f710 2024-03-25T17:48:11.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.700+0000 7fdb04ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:11.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.700+0000 7fdb04ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:48:11.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.700+0000 7fdb04ff9640 10 monclient(hunting): dump: 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:11.701 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:11.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.707+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3500305546 0 0) 0x7fdaf4004eb0 con 0x7fdb080a98f0 2024-03-25T17:48:11.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.707+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb081398a0 con 0x7fdb080a98f0 2024-03-25T17:48:11.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.707+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdaf8001ae0 con 0x7fdb080a98f0 2024-03-25T17:48:11.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.707+0000 7fdb04ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.707+0000 7fdb04ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.707+0000 7fdb04ff9640 10 monclient(hunting): dump: 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:11.708 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3677382342 0 0) 0x7fdb081d4c90 con 0x7fdb081430c0 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 >> v1:172.21.15.44:6790/0 conn(0x7fdb0813f710 legacy=0x7fdb080aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 >> v1:172.21.15.44:6789/0 conn(0x7fdb080a98f0 legacy=0x7fdb081d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 1 monclient: found mon.b 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb080b16b0 con 0x7fdb081430c0 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 10 monclient: _finish_auth 0 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:48:11.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 5 monclient: authenticate success, global_id 4621 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 10 monclient: _renew_subs 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb080b0c10 con 0x7fdb081430c0 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 10 monclient: _renew_subs 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb080b07f0 con 0x7fdb081430c0 2024-03-25T17:48:11.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.710+0000 7fdb0e70f640 1 librados: init done 2024-03-25T17:48:11.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.711+0000 7fdb0e70f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:11.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.711+0000 7fdb0e70f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:11.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.711+0000 7fdb0e70f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:11.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:11.711+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb081d4e20 con 0x7fdb081430c0 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.002+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb00003970 con 0x7fdb081430c0 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.002+0000 7fdb04ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.002+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fdb00003d80 con 0x7fdb081430c0 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.002+0000 7fdb04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.002+0000 7fdb04ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.002+0000 7fdb04ff9640 10 monclient: dump: 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:12.003 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:12.004 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:12.004 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:12.004 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:12.004 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:12.004 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:12.004 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:12.004 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:12.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.002+0000 7fdb067fc640 4 set_mon_vals no callback set 2024-03-25T17:48:12.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.003+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fdb00003fa0 con 0x7fdb081430c0 2024-03-25T17:48:12.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.004+0000 7fdb07fff640 10 monclient: get_auth_request con 0x7fdaf406e160 auth_method 0 2024-03-25T17:48:12.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.004+0000 7fdb04ff9640 10 client.4621.objecter ms_handle_connect 0x7fdaf406e160 2024-03-25T17:48:12.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.008+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fdb00054aa0 con 0x7fdb081430c0 2024-03-25T17:48:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.008+0000 7fdb04ff9640 10 client.4621.objecter ms_dispatch 0x7fdb08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:48:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.008+0000 7fdb04ff9640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:48:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.008+0000 7fdb04ff9640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:48:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.008+0000 7fdb04ff9640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T17:48:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.008+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb000505b0 con 0x7fdb081430c0 2024-03-25T17:48:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.008+0000 7fdb04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.008+0000 7fdb04ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:48:12.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.247+0000 7fdb0e70f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:12.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.247+0000 7fdb0e70f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:12.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.247+0000 7fdb0e70f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:12.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.247+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb080627b0 con 0x7fdb081430c0 2024-03-25T17:48:12.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.323+0000 7fdb04ff9640 1 -- 172.21.15.44:0/2487309123 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fdb000524e0 con 0x7fdb081430c0 2024-03-25T17:48:12.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.323+0000 7fdb04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:12.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.323+0000 7fdb04ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:48:12.324 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:48:12.324 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:48:12.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 10 librados: watch_flush enter 2024-03-25T17:48:12.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 10 librados: watch_flush exit 2024-03-25T17:48:12.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T17:48:12.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 10 client.4621.objecter successfully canceled tick 2024-03-25T17:48:12.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 >> v1:172.21.15.44:6800/3257043132 conn(0x7fdaf406e160 legacy=0x7fdaf4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:12.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 10 monclient: shutdown 2024-03-25T17:48:12.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:12.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 >> v1:172.21.15.111:6789/0 conn(0x7fdb081430c0 legacy=0x7fdb081d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:12.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 shutdown_connections 2024-03-25T17:48:12.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 >> 172.21.15.44:0/2487309123 conn(0x7fdb080b7ea0 msgr2=0x7fdb08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:12.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 shutdown_connections 2024-03-25T17:48:12.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 1 -- 172.21.15.44:0/2487309123 wait complete. 2024-03-25T17:48:12.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:12.325+0000 7fdb0e70f640 1 librados: shutdown 2024-03-25T17:48:12.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:48:19.078 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:36.434 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.086 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:36.442 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.122 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:36.477 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.134 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:36.454 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.140 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:36.496 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.141 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:36.461 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.144 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:36.499 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.145 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:48:19.174 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:36.493 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.183 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:36.539 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:19.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.247+0000 7f052b500640 10 monclient: get_monmap_and_config 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.247+0000 7f052b500640 10 monclient: build_initial_monmap 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.247+0000 7f052b500640 10 monclient: monmap: 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:48:19.248464+0000 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:48:19.248464+0000 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:48:19.248 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:19.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 1 Processor -- start 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 1 -- start start 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 10 monclient: init 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 10 monclient(hunting): picked mon.noname-c con 0x7f05240b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 10 monclient(hunting): picked mon.noname-a con 0x7f05240b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 10 monclient(hunting): picked mon.noname-b con 0x7f052414be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05240af940 con 0x7f05240b1730 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05240b0a70 con 0x7f05240b4d60 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05240b0d50 con 0x7f052414be90 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:19.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.248+0000 7f052b500640 10 monclient(hunting): authenticate will time out at 6502.476610s 2024-03-25T17:48:19.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.255+0000 7f0529a9d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f052414be90 0x7f052414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:47818/0 (socket says 172.21.15.44:47818) 2024-03-25T17:48:19.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.255+0000 7f0529a9d640 1 -- 172.21.15.44:0/2370492354 learned_addr learned my addr 172.21.15.44:0/2370492354 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:19.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.353+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075930060 0 0) 0x7f05240b0a70 con 0x7f05240b4d60 2024-03-25T17:48:19.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.353+0000 7f051bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:48:19.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.353+0000 7f051bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:19.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.353+0000 7f051bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:19.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.353+0000 7f051bfff640 10 monclient(hunting): my global_id is 4624 2024-03-25T17:48:19.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.353+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0510003520 con 0x7f05240b4d60 2024-03-25T17:48:19.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.354+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512380089 0 0) 0x7f05240b0d50 con 0x7f052414be90 2024-03-25T17:48:19.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.354+0000 7f051bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:48:19.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.354+0000 7f051bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:19.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.354+0000 7f051bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:19.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.354+0000 7f051bfff640 10 monclient(hunting): my global_id is 4631 2024-03-25T17:48:19.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.354+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05240b0a70 con 0x7f052414be90 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.377+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3916170894 0 0) 0x7f0510003520 con 0x7f05240b4d60 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.377+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 >> v1:172.21.15.44:6790/0 conn(0x7f052414be90 legacy=0x7f052414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.377+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 >> v1:172.21.15.44:6789/0 conn(0x7f05240b1730 legacy=0x7f05240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.377+0000 7f051bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.377+0000 7f051bfff640 1 monclient: found mon.noname-c 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.377+0000 7f051bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.377+0000 7f051bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.377+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05240b1030 con 0x7f05240b4d60 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f051bfff640 10 monclient: _finish_auth 0 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f051bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f051bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0520001690 con 0x7f05240b4d60 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f051bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:19.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f051bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f051bfff640 10 monclient: dump: 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f052b500640 5 monclient: authenticate success, global_id 4624 2024-03-25T17:48:19.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.378+0000 7f052b500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.410+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05200017f0 con 0x7f05240b4d60 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.410+0000 7f051bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.410+0000 7f051bfff640 1 -- 172.21.15.44:0/2370492354 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0520001c00 con 0x7f05240b4d60 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.410+0000 7f052b500640 10 monclient: get_monmap_and_config success 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.410+0000 7f052b500640 4 set_mon_vals no callback set 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.410+0000 7f051bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.410+0000 7f051bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.410+0000 7f051bfff640 10 monclient: dump: 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:19.411 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:19.412 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:19.412 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:19.412 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:19.412 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:19.412 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:19.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.411+0000 7f052b500640 10 monclient: shutdown 2024-03-25T17:48:19.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.411+0000 7f052b500640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:19.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.411+0000 7f052b500640 1 -- 172.21.15.44:0/2370492354 >> v1:172.21.15.111:6789/0 conn(0x7f05240b4d60 legacy=0x7f052413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:19.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.411+0000 7f052b500640 1 -- 172.21.15.44:0/2370492354 shutdown_connections 2024-03-25T17:48:19.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.411+0000 7f052b500640 1 -- 172.21.15.44:0/2370492354 >> 172.21.15.44:0/2370492354 conn(0x7f05240617a0 msgr2=0x7f0524061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:19.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.411+0000 7f052b500640 1 -- 172.21.15.44:0/2370492354 shutdown_connections 2024-03-25T17:48:19.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.411+0000 7f052b500640 1 -- 172.21.15.44:0/2370492354 wait complete. 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.412+0000 7f052b500640 10 monclient: build_initial_monmap 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.412+0000 7f052b500640 10 monclient: monmap: 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.413+0000 7f052b500640 1 librados: starting msgr at 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.413+0000 7f052b500640 1 librados: starting objecter 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.413+0000 7f052b500640 1 Processor -- start 2024-03-25T17:48:19.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.413+0000 7f052b500640 1 -- start start 2024-03-25T17:48:19.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.413+0000 7f052b500640 1 librados: setting wanted keys 2024-03-25T17:48:19.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.413+0000 7f052b500640 1 librados: calling monclient init 2024-03-25T17:48:19.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.413+0000 7f052b500640 10 monclient: init 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 10 monclient(hunting): picked mon.noname-b con 0x7f05240b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 10 monclient(hunting): picked mon.noname-a con 0x7f05240b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 10 monclient(hunting): picked mon.noname-c con 0x7f052414be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0524140d00 con 0x7f05240b4d60 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0524141cc0 con 0x7f05240b1730 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05241d8470 con 0x7f052414be90 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:19.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.414+0000 7f052b500640 10 monclient(hunting): authenticate will time out at 6502.641933s 2024-03-25T17:48:19.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.429+0000 7f0528a9b640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f05240b4d60 0x7f05241d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56438/0 (socket says 172.21.15.44:56438) 2024-03-25T17:48:19.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.429+0000 7f0528a9b640 1 -- 172.21.15.44:0/3421745080 learned_addr learned my addr 172.21.15.44:0/3421745080 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:19.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.433+0000 7f0519ffb640 10 client.?.objecter ms_handle_connect 0x7f05240b4d60 2024-03-25T17:48:19.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.433+0000 7f0519ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:19.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.449+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964825853 0 0) 0x7f0524140d00 con 0x7f05240b4d60 2024-03-25T17:48:19.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.449+0000 7f0519ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:48:19.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.449+0000 7f0519ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:19.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.449+0000 7f0519ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:19.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.449+0000 7f0519ffb640 10 monclient(hunting): my global_id is 4638 2024-03-25T17:48:19.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.449+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0508004eb0 con 0x7f05240b4d60 2024-03-25T17:48:19.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.449+0000 7f0519ffb640 10 client.?.objecter ms_handle_connect 0x7f05240b1730 2024-03-25T17:48:19.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.449+0000 7f0519ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:19.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.453+0000 7f0519ffb640 10 client.?.objecter ms_handle_connect 0x7f052414be90 2024-03-25T17:48:19.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.453+0000 7f0519ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:19.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.459+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 342727258 0 0) 0x7f0508004eb0 con 0x7f05240b4d60 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.459+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0524140d00 con 0x7f05240b4d60 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.459+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0514001a30 con 0x7f05240b4d60 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.459+0000 7f0519ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.459+0000 7f0519ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.459+0000 7f0519ffb640 10 monclient(hunting): dump: 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:19.460 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:19.461 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:19.461 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:19.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.459+0000 7f0519ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:48:19.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.464+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171985030 0 0) 0x7f0524141cc0 con 0x7f05240b1730 2024-03-25T17:48:19.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.464+0000 7f0519ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:48:19.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.464+0000 7f0519ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:19.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.464+0000 7f0519ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:19.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.464+0000 7f0519ffb640 10 monclient(hunting): my global_id is 4627 2024-03-25T17:48:19.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.464+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0508004eb0 con 0x7f05240b1730 2024-03-25T17:48:19.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633939730 0 0) 0x7f0524140d00 con 0x7f05240b4d60 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 >> v1:172.21.15.44:6790/0 conn(0x7f052414be90 legacy=0x7f0524134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 >> v1:172.21.15.111:6789/0 conn(0x7f05240b1730 legacy=0x7f05241405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 1 monclient: found mon.a 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05241d8600 con 0x7f05240b4d60 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 10 monclient: _finish_auth 0 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f0519ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 5 monclient: authenticate success, global_id 4638 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 10 monclient: _renew_subs 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:19.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052413b3c0 con 0x7f05240b4d60 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 10 client.4638.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 10 monclient: _renew_subs 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05241425a0 con 0x7f05240b4d60 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.469+0000 7f052b500640 1 librados: init done 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.470+0000 7f052b500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.470+0000 7f052b500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.470+0000 7f052b500640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:19.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.470+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05240b2270 con 0x7f05240b4d60 2024-03-25T17:48:19.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.784+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0514001cb0 con 0x7f05240b4d60 2024-03-25T17:48:19.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.784+0000 7f0519ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.784+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0514003a60 con 0x7f05240b4d60 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.785+0000 7f0519ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.785+0000 7f0519ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.785+0000 7f0519ffb640 10 monclient: dump: 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:19.786 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:19.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.785+0000 7f051b7fe640 4 set_mon_vals no callback set 2024-03-25T17:48:19.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.785+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f0514003c40 con 0x7f05240b4d60 2024-03-25T17:48:19.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.786+0000 7f052929c640 10 monclient: get_auth_request con 0x7f050806dd90 auth_method 0 2024-03-25T17:48:19.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.786+0000 7f0519ffb640 10 client.4638.objecter ms_handle_connect 0x7f050806dd90 2024-03-25T17:48:19.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.792+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f0514054880 con 0x7f05240b4d60 2024-03-25T17:48:19.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.792+0000 7f0519ffb640 10 client.4638.objecter ms_dispatch 0x7f0524000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:48:19.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.792+0000 7f0519ffb640 3 client.4638.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:48:19.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.792+0000 7f0519ffb640 3 client.4638.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:48:19.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.792+0000 7f0519ffb640 20 client.4638.objecter dump_active .. 0 homeless 2024-03-25T17:48:19.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.793+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0514050390 con 0x7f05240b4d60 2024-03-25T17:48:19.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.793+0000 7f0519ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:19.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:19.793+0000 7f0519ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:48:20.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.048+0000 7f052b500640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:20.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.049+0000 7f052b500640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:20.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.049+0000 7f052b500640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:48:20.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.049+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05240627b0 con 0x7f05240b4d60 2024-03-25T17:48:20.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.078+0000 7f0519ffb640 1 -- 172.21.15.44:0/3421745080 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f05140522c0 con 0x7f05240b4d60 2024-03-25T17:48:20.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.078+0000 7f0519ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:20.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.078+0000 7f0519ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:48:20.079 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:48:20.079 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 10 librados: watch_flush enter 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 10 librados: watch_flush exit 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 20 client.4638.objecter shutdown clearing up homeless session... 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 10 client.4638.objecter successfully canceled tick 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 >> v1:172.21.15.44:6800/3257043132 conn(0x7f050806dd90 legacy=0x7f0508070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 10 monclient: shutdown 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 >> v1:172.21.15.44:6789/0 conn(0x7f05240b4d60 legacy=0x7f05241d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 shutdown_connections 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 >> 172.21.15.44:0/3421745080 conn(0x7f05240617a0 msgr2=0x7f05240b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:20.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 shutdown_connections 2024-03-25T17:48:20.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 1 -- 172.21.15.44:0/3421745080 wait complete. 2024-03-25T17:48:20.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:20.080+0000 7f052b500640 1 librados: shutdown 2024-03-25T17:48:20.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:48:26.899 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.012+0000 7fe7defd6640 10 monclient: get_monmap_and_config 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.012+0000 7fe7defd6640 10 monclient: build_initial_monmap 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.012+0000 7fe7defd6640 10 monclient: monmap: 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:48:27.013817+0000 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:48:27.013817+0000 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:27.013 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:27.014 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:27.014 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:48:27.014 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:48:27.014 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.013+0000 7fe7defd6640 1 Processor -- start 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.013+0000 7fe7defd6640 1 -- start start 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.013+0000 7fe7defd6640 10 monclient: init 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d813f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:27.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d8142520 con 0x7fe7d813f710 2024-03-25T17:48:27.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80b28a0 con 0x7fe7d80a98f0 2024-03-25T17:48:27.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80b2a30 con 0x7fe7d81430c0 2024-03-25T17:48:27.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:27.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.014+0000 7fe7defd6640 10 monclient(hunting): authenticate will time out at 6510.242118s 2024-03-25T17:48:27.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.019+0000 7fe7dcd72640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe7d80a98f0 0x7fe7d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:33310/0 (socket says 172.21.15.44:33310) 2024-03-25T17:48:27.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.019+0000 7fe7dcd72640 1 -- 172.21.15.44:0/665013094 learned_addr learned my addr 172.21.15.44:0/665013094 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:27.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.024+0000 7fe7d77fe640 1 -- 172.21.15.44:0/665013094 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129358234 0 0) 0x7fe7d80b28a0 con 0x7fe7d80a98f0 2024-03-25T17:48:27.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.024+0000 7fe7d77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:48:27.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.024+0000 7fe7d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:27.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.024+0000 7fe7d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:27.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.024+0000 7fe7d77fe640 10 monclient(hunting): my global_id is 4630 2024-03-25T17:48:27.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.024+0000 7fe7d77fe640 1 -- 172.21.15.44:0/665013094 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c8003520 con 0x7fe7d80a98f0 2024-03-25T17:48:27.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 1 -- 172.21.15.44:0/665013094 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1081224586 0 0) 0x7fe7c8003520 con 0x7fe7d80a98f0 2024-03-25T17:48:27.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 1 -- 172.21.15.44:0/665013094 >> v1:172.21.15.44:6790/0 conn(0x7fe7d81430c0 legacy=0x7fe7d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:48:27.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 1 -- 172.21.15.44:0/665013094 >> v1:172.21.15.44:6789/0 conn(0x7fe7d813f710 legacy=0x7fe7d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:48:27.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 10 monclient: _finish_hunting 0 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 1 monclient: found mon.noname-c 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 1 -- 172.21.15.44:0/665013094 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80b44c0 con 0x7fe7d80a98f0 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 10 monclient: _finish_auth 0 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 10 monclient: _check_auth_tickets 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 1 -- 172.21.15.44:0/665013094 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe7cc001690 con 0x7fe7d80a98f0 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7defd6640 5 monclient: authenticate success, global_id 4630 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7defd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7d77fe640 10 monclient: dump: 2024-03-25T17:48:27.030 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:27.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.028+0000 7fe7defd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:27.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.036+0000 7fe7d77fe640 1 -- 172.21.15.44:0/665013094 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7cc001860 con 0x7fe7d80a98f0 2024-03-25T17:48:27.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.036+0000 7fe7d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:27.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.036+0000 7fe7defd6640 10 monclient: get_monmap_and_config success 2024-03-25T17:48:27.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.036+0000 7fe7defd6640 4 set_mon_vals no callback set 2024-03-25T17:48:27.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.037+0000 7fe7defd6640 10 monclient: shutdown 2024-03-25T17:48:27.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.037+0000 7fe7defd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:27.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.037+0000 7fe7defd6640 1 -- 172.21.15.44:0/665013094 >> v1:172.21.15.111:6789/0 conn(0x7fe7d80a98f0 legacy=0x7fe7d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:27.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.037+0000 7fe7defd6640 1 -- 172.21.15.44:0/665013094 shutdown_connections 2024-03-25T17:48:27.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.037+0000 7fe7defd6640 1 -- 172.21.15.44:0/665013094 >> 172.21.15.44:0/665013094 conn(0x7fe7d80b7ea0 msgr2=0x7fe7d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:27.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.037+0000 7fe7defd6640 1 -- 172.21.15.44:0/665013094 shutdown_connections 2024-03-25T17:48:27.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.037+0000 7fe7defd6640 1 -- 172.21.15.44:0/665013094 wait complete. 2024-03-25T17:48:27.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.038+0000 7fe7defd6640 10 monclient: build_initial_monmap 2024-03-25T17:48:27.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.038+0000 7fe7defd6640 10 monclient: monmap: 2024-03-25T17:48:27.039 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:27.039 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:48:27.039 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:48:27.039 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:48:27.039 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.038+0000 7fe7defd6640 1 librados: starting msgr at 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.038+0000 7fe7defd6640 1 librados: starting objecter 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.039+0000 7fe7defd6640 1 Processor -- start 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.039+0000 7fe7defd6640 1 -- start start 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.039+0000 7fe7defd6640 1 librados: setting wanted keys 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.039+0000 7fe7defd6640 1 librados: calling monclient init 2024-03-25T17:48:27.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.039+0000 7fe7defd6640 10 monclient: init 2024-03-25T17:48:27.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:27.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:48:27.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:27.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d813f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:27.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:27.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80af710 con 0x7fe7d813f710 2024-03-25T17:48:27.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80b1730 con 0x7fe7d80a98f0 2024-03-25T17:48:27.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80b05b0 con 0x7fe7d81430c0 2024-03-25T17:48:27.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:27.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.040+0000 7fe7defd6640 10 monclient(hunting): authenticate will time out at 6510.267913s 2024-03-25T17:48:27.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.045+0000 7fe7dd573640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fe7d81430c0 0x7fe7d81e1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:58988/0 (socket says 172.21.15.44:58988) 2024-03-25T17:48:27.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.045+0000 7fe7dd573640 1 -- 172.21.15.44:0/15730590 learned_addr learned my addr 172.21.15.44:0/15730590 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:27.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.047+0000 7fe7d57fa640 10 client.?.objecter ms_handle_connect 0x7fe7d81430c0 2024-03-25T17:48:27.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.047+0000 7fe7d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:27.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.058+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879228352 0 0) 0x7fe7d80b05b0 con 0x7fe7d81430c0 2024-03-25T17:48:27.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.058+0000 7fe7d57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:48:27.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.058+0000 7fe7d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:27.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.058+0000 7fe7d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:27.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.058+0000 7fe7d57fa640 10 monclient(hunting): my global_id is 4637 2024-03-25T17:48:27.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.058+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc004eb0 con 0x7fe7d81430c0 2024-03-25T17:48:27.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.069+0000 7fe7d57fa640 10 client.?.objecter ms_handle_connect 0x7fe7d80a98f0 2024-03-25T17:48:27.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.069+0000 7fe7d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:27.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.077+0000 7fe7d57fa640 10 client.?.objecter ms_handle_connect 0x7fe7d813f710 2024-03-25T17:48:27.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.078+0000 7fe7d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:27.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.078+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928188026 0 0) 0x7fe7bc004eb0 con 0x7fe7d81430c0 2024-03-25T17:48:27.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.078+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7d80b05b0 con 0x7fe7d81430c0 2024-03-25T17:48:27.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.078+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe7d0001ad0 con 0x7fe7d81430c0 2024-03-25T17:48:27.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.078+0000 7fe7d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:27.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.078+0000 7fe7d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.078+0000 7fe7d57fa640 10 monclient(hunting): dump: 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:27.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.078+0000 7fe7d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:48:27.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.083+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144817410 0 0) 0x7fe7d80b1730 con 0x7fe7d80a98f0 2024-03-25T17:48:27.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.083+0000 7fe7d57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:48:27.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.083+0000 7fe7d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:27.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.083+0000 7fe7d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:27.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.083+0000 7fe7d57fa640 10 monclient(hunting): my global_id is 4633 2024-03-25T17:48:27.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.083+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc004eb0 con 0x7fe7d80a98f0 2024-03-25T17:48:27.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.096+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107489073 0 0) 0x7fe7d80af710 con 0x7fe7d813f710 2024-03-25T17:48:27.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.096+0000 7fe7d57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:48:27.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.096+0000 7fe7d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:27.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.096+0000 7fe7d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:27.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.096+0000 7fe7d57fa640 10 monclient(hunting): my global_id is 4641 2024-03-25T17:48:27.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.096+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d80b1730 con 0x7fe7d813f710 2024-03-25T17:48:27.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2052326828 0 0) 0x7fe7d80b05b0 con 0x7fe7d81430c0 2024-03-25T17:48:27.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 >> v1:172.21.15.111:6789/0 conn(0x7fe7d80a98f0 legacy=0x7fe7d80af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:27.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 >> v1:172.21.15.44:6789/0 conn(0x7fe7d813f710 legacy=0x7fe7d80b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:27.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 10 monclient: _finish_hunting 0 2024-03-25T17:48:27.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 1 monclient: found mon.c 2024-03-25T17:48:27.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80b0890 con 0x7fe7d81430c0 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 10 monclient: _finish_auth 0 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 10 monclient: _check_auth_tickets 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 5 monclient: authenticate success, global_id 4637 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 10 monclient: _renew_subs 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d80aeae0 con 0x7fe7d81430c0 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 10 monclient: _renew_subs 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d80ab900 con 0x7fe7d81430c0 2024-03-25T17:48:27.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.104+0000 7fe7defd6640 1 librados: init done 2024-03-25T17:48:27.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.105+0000 7fe7defd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:27.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.106+0000 7fe7defd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:27.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.106+0000 7fe7defd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:27.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.106+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d80ae930 con 0x7fe7d81430c0 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.459+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7d0002020 con 0x7fe7d81430c0 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.459+0000 7fe7d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.459+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe7d0004d80 con 0x7fe7d81430c0 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.459+0000 7fe7d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.459+0000 7fe7d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.459+0000 7fe7d57fa640 10 monclient: dump: 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:27.460 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:27.461 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:27.461 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:27.461 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:27.461 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:27.461 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:27.461 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:27.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.459+0000 7fe7d6ffd640 4 set_mon_vals no callback set 2024-03-25T17:48:27.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.460+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fe7d001ad50 con 0x7fe7d81430c0 2024-03-25T17:48:27.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.461+0000 7fe7dcd72640 10 monclient: get_auth_request con 0x7fe7bc06de60 auth_method 0 2024-03-25T17:48:27.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.461+0000 7fe7d57fa640 10 client.4637.objecter ms_handle_connect 0x7fe7bc06de60 2024-03-25T17:48:27.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.467+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fe7d0054a70 con 0x7fe7d81430c0 2024-03-25T17:48:27.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.467+0000 7fe7d57fa640 10 client.4637.objecter ms_dispatch 0x7fe7d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:48:27.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.467+0000 7fe7d57fa640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:48:27.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.467+0000 7fe7d57fa640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:48:27.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.467+0000 7fe7d57fa640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-25T17:48:27.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.467+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7d00503b0 con 0x7fe7d81430c0 2024-03-25T17:48:27.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.467+0000 7fe7d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:27.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.467+0000 7fe7d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:48:27.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.728+0000 7fe7defd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:27.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.728+0000 7fe7defd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:27.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.728+0000 7fe7defd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:27.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.728+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7d80aec70 con 0x7fe7d81430c0 2024-03-25T17:48:27.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.755+0000 7fe7d57fa640 1 -- 172.21.15.44:0/15730590 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fe7d00524b0 con 0x7fe7d81430c0 2024-03-25T17:48:27.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.755+0000 7fe7d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:27.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.755+0000 7fe7d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:48:27.756 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:48:27.756 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:48:27.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 10 librados: watch_flush enter 2024-03-25T17:48:27.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 10 librados: watch_flush exit 2024-03-25T17:48:27.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-25T17:48:27.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 10 client.4637.objecter successfully canceled tick 2024-03-25T17:48:27.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 >> v1:172.21.15.44:6800/3257043132 conn(0x7fe7bc06de60 legacy=0x7fe7bc070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:27.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 10 monclient: shutdown 2024-03-25T17:48:27.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:27.759 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 >> v1:172.21.15.44:6790/0 conn(0x7fe7d81430c0 legacy=0x7fe7d81e1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:27.759 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 shutdown_connections 2024-03-25T17:48:27.759 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 >> 172.21.15.44:0/15730590 conn(0x7fe7d80b7ea0 msgr2=0x7fe7d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:27.759 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 shutdown_connections 2024-03-25T17:48:27.759 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 1 -- 172.21.15.44:0/15730590 wait complete. 2024-03-25T17:48:27.759 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:27.757+0000 7fe7defd6640 1 librados: shutdown 2024-03-25T17:48:27.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:48:34.576 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.696+0000 7efd6b9cf640 10 monclient: get_monmap_and_config 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.696+0000 7efd6b9cf640 10 monclient: build_initial_monmap 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.696+0000 7efd6b9cf640 10 monclient: monmap: 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:48:34.697190+0000 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:48:34.697190+0000 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:48:34.697 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:48:34.698 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:34.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.696+0000 7efd6b9cf640 1 Processor -- start 2024-03-25T17:48:34.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.697+0000 7efd6b9cf640 1 -- start start 2024-03-25T17:48:34.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.697+0000 7efd6b9cf640 10 monclient: init 2024-03-25T17:48:34.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:34.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:48:34.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 10 monclient(hunting): picked mon.noname-a con 0x7efd640a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:34.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 10 monclient(hunting): picked mon.noname-b con 0x7efd6413f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:34.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 10 monclient(hunting): picked mon.noname-c con 0x7efd641430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:34.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64142520 con 0x7efd640a98f0 2024-03-25T17:48:34.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640b28a0 con 0x7efd641430c0 2024-03-25T17:48:34.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640b2a30 con 0x7efd6413f710 2024-03-25T17:48:34.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:34.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.698+0000 7efd6b9cf640 10 monclient(hunting): authenticate will time out at 6517.925838s 2024-03-25T17:48:34.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.709+0000 7efd69f6c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efd641430c0 0x7efd6414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:47202/0 (socket says 172.21.15.44:47202) 2024-03-25T17:48:34.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.709+0000 7efd69f6c640 1 -- 172.21.15.44:0/4137301771 learned_addr learned my addr 172.21.15.44:0/4137301771 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:34.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.719+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396182477 0 0) 0x7efd64142520 con 0x7efd640a98f0 2024-03-25T17:48:34.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.719+0000 7efd5bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:48:34.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.719+0000 7efd5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:34.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.719+0000 7efd5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:34.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.719+0000 7efd5bfff640 10 monclient(hunting): my global_id is 4644 2024-03-25T17:48:34.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.719+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd50003520 con 0x7efd640a98f0 2024-03-25T17:48:34.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.724+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149185579 0 0) 0x7efd640b28a0 con 0x7efd641430c0 2024-03-25T17:48:34.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.724+0000 7efd5bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:48:34.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.724+0000 7efd5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:34.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.724+0000 7efd5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:34.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.724+0000 7efd5bfff640 10 monclient(hunting): my global_id is 4636 2024-03-25T17:48:34.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.724+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd64142520 con 0x7efd641430c0 2024-03-25T17:48:34.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953179748 0 0) 0x7efd50003520 con 0x7efd640a98f0 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 >> v1:172.21.15.44:6790/0 conn(0x7efd6413f710 legacy=0x7efd641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 >> v1:172.21.15.111:6789/0 conn(0x7efd641430c0 legacy=0x7efd6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 1 monclient: found mon.noname-a 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640b44c0 con 0x7efd640a98f0 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 10 monclient: _finish_auth 0 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efd60001690 con 0x7efd640a98f0 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd6b9cf640 5 monclient: authenticate success, global_id 4644 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd6b9cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:34.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd5bfff640 10 monclient: dump: 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:34.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.728+0000 7efd6b9cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:34.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.735+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd600017f0 con 0x7efd640a98f0 2024-03-25T17:48:34.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.735+0000 7efd5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:34.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.735+0000 7efd5bfff640 1 -- 172.21.15.44:0/4137301771 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efd60001c00 con 0x7efd640a98f0 2024-03-25T17:48:34.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.735+0000 7efd5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:34.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.735+0000 7efd5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.735+0000 7efd5bfff640 10 monclient: dump: 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.735+0000 7efd6b9cf640 10 monclient: get_monmap_and_config success 2024-03-25T17:48:34.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.735+0000 7efd6b9cf640 4 set_mon_vals no callback set 2024-03-25T17:48:34.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.736+0000 7efd6b9cf640 10 monclient: shutdown 2024-03-25T17:48:34.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.736+0000 7efd6b9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:34.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.736+0000 7efd6b9cf640 1 -- 172.21.15.44:0/4137301771 >> v1:172.21.15.44:6789/0 conn(0x7efd640a98f0 legacy=0x7efd640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:34.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.736+0000 7efd6b9cf640 1 -- 172.21.15.44:0/4137301771 shutdown_connections 2024-03-25T17:48:34.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.736+0000 7efd6b9cf640 1 -- 172.21.15.44:0/4137301771 >> 172.21.15.44:0/4137301771 conn(0x7efd640b7ea0 msgr2=0x7efd640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.736+0000 7efd6b9cf640 1 -- 172.21.15.44:0/4137301771 shutdown_connections 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.736+0000 7efd6b9cf640 1 -- 172.21.15.44:0/4137301771 wait complete. 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 10 monclient: build_initial_monmap 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 10 monclient: monmap: 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:48:34.738 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:34.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 1 librados: starting msgr at 2024-03-25T17:48:34.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 1 librados: starting objecter 2024-03-25T17:48:34.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 1 Processor -- start 2024-03-25T17:48:34.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 1 -- start start 2024-03-25T17:48:34.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 1 librados: setting wanted keys 2024-03-25T17:48:34.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 1 librados: calling monclient init 2024-03-25T17:48:34.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.737+0000 7efd6b9cf640 10 monclient: init 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 10 monclient(hunting): picked mon.noname-c con 0x7efd640a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 10 monclient(hunting): picked mon.noname-a con 0x7efd6413f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 10 monclient(hunting): picked mon.noname-b con 0x7efd641430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd641d7910 con 0x7efd6413f710 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640abfe0 con 0x7efd641430c0 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd641d84b0 con 0x7efd640a98f0 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:34.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.738+0000 7efd6b9cf640 10 monclient(hunting): authenticate will time out at 6517.966160s 2024-03-25T17:48:34.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.750+0000 7efd69f6c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efd641430c0 0x7efd64138d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:47226/0 (socket says 172.21.15.44:47226) 2024-03-25T17:48:34.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.750+0000 7efd69f6c640 1 -- 172.21.15.44:0/488541602 learned_addr learned my addr 172.21.15.44:0/488541602 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:34.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.750+0000 7efd59ffb640 10 client.?.objecter ms_handle_connect 0x7efd641430c0 2024-03-25T17:48:34.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.750+0000 7efd59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:34.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.754+0000 7efd59ffb640 10 client.?.objecter ms_handle_connect 0x7efd640a98f0 2024-03-25T17:48:34.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.754+0000 7efd59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:34.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.755+0000 7efd59ffb640 10 client.?.objecter ms_handle_connect 0x7efd6413f710 2024-03-25T17:48:34.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.755+0000 7efd59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:34.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.760+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031996653 0 0) 0x7efd640abfe0 con 0x7efd641430c0 2024-03-25T17:48:34.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.760+0000 7efd59ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:48:34.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.760+0000 7efd59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:34.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.760+0000 7efd59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:34.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.760+0000 7efd59ffb640 10 monclient(hunting): my global_id is 4639 2024-03-25T17:48:34.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.760+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd48004eb0 con 0x7efd641430c0 2024-03-25T17:48:34.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.764+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029057136 0 0) 0x7efd641d7910 con 0x7efd6413f710 2024-03-25T17:48:34.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.764+0000 7efd59ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:48:34.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.764+0000 7efd59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:34.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.764+0000 7efd59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:34.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.764+0000 7efd59ffb640 10 monclient(hunting): my global_id is 4647 2024-03-25T17:48:34.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.764+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd640abfe0 con 0x7efd6413f710 2024-03-25T17:48:34.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.771+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1324588094 0 0) 0x7efd48004eb0 con 0x7efd641430c0 2024-03-25T17:48:34.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.771+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd641d7910 con 0x7efd641430c0 2024-03-25T17:48:34.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.771+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efd5c001c10 con 0x7efd641430c0 2024-03-25T17:48:34.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.771+0000 7efd59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:34.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.771+0000 7efd59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.771+0000 7efd59ffb640 10 monclient(hunting): dump: 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:34.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.771+0000 7efd59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:48:34.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.776+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3278810093 0 0) 0x7efd640abfe0 con 0x7efd6413f710 2024-03-25T17:48:34.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.776+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd48004eb0 con 0x7efd6413f710 2024-03-25T17:48:34.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.776+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efd540018b0 con 0x7efd6413f710 2024-03-25T17:48:34.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.776+0000 7efd59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:34.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.776+0000 7efd59ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:48:34.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.776+0000 7efd59ffb640 10 monclient(hunting): dump: 2024-03-25T17:48:34.777 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:34.778 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:34.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.782+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834834384 0 0) 0x7efd641d84b0 con 0x7efd640a98f0 2024-03-25T17:48:34.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.782+0000 7efd59ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:48:34.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.782+0000 7efd59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:34.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.782+0000 7efd59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:34.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.782+0000 7efd59ffb640 10 monclient(hunting): my global_id is 4643 2024-03-25T17:48:34.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.782+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd640abfe0 con 0x7efd640a98f0 2024-03-25T17:48:34.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2419309174 0 0) 0x7efd641d7910 con 0x7efd641430c0 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 >> v1:172.21.15.44:6790/0 conn(0x7efd640a98f0 legacy=0x7efd641d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 >> v1:172.21.15.44:6789/0 conn(0x7efd6413f710 legacy=0x7efd641385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 1 monclient: found mon.b 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd641d51f0 con 0x7efd641430c0 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 10 monclient: _finish_auth 0 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd6b9cf640 5 monclient: authenticate success, global_id 4639 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd6b9cf640 10 monclient: _renew_subs 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd6b9cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:34.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.787+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd641e2ed0 con 0x7efd641430c0 2024-03-25T17:48:34.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.788+0000 7efd6b9cf640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:48:34.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.788+0000 7efd6b9cf640 10 monclient: _renew_subs 2024-03-25T17:48:34.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.788+0000 7efd6b9cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:34.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.788+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd641e3240 con 0x7efd641430c0 2024-03-25T17:48:34.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.788+0000 7efd6b9cf640 1 librados: init done 2024-03-25T17:48:34.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.789+0000 7efd6b9cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:34.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.789+0000 7efd6b9cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:34.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.789+0000 7efd6b9cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:34.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:34.789+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd640627b0 con 0x7efd641430c0 2024-03-25T17:48:35.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.068+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd5c004b60 con 0x7efd641430c0 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.068+0000 7efd59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.068+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7efd5c004e70 con 0x7efd641430c0 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.068+0000 7efd59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.068+0000 7efd59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.068+0000 7efd59ffb640 10 monclient: dump: 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:35.070 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:35.071 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:35.071 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:35.071 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:35.071 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:35.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.068+0000 7efd5b7fe640 4 set_mon_vals no callback set 2024-03-25T17:48:35.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.069+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7efd5c01ae40 con 0x7efd641430c0 2024-03-25T17:48:35.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.069+0000 7efd6976b640 10 monclient: get_auth_request con 0x7efd4806df00 auth_method 0 2024-03-25T17:48:35.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.070+0000 7efd59ffb640 10 client.4639.objecter ms_handle_connect 0x7efd4806df00 2024-03-25T17:48:35.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.078+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7efd5c054ab0 con 0x7efd641430c0 2024-03-25T17:48:35.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.079+0000 7efd59ffb640 10 client.4639.objecter ms_dispatch 0x7efd64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:48:35.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.079+0000 7efd59ffb640 3 client.4639.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:48:35.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.079+0000 7efd59ffb640 3 client.4639.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:48:35.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.079+0000 7efd59ffb640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T17:48:35.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.080+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd5c0505c0 con 0x7efd641430c0 2024-03-25T17:48:35.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.080+0000 7efd59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:35.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.080+0000 7efd59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:48:35.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.322+0000 7efd6b9cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:35.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.322+0000 7efd6b9cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:35.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.322+0000 7efd6b9cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:35.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.322+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd641d7ac0 con 0x7efd641430c0 2024-03-25T17:48:35.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.351+0000 7efd59ffb640 1 -- 172.21.15.44:0/488541602 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7efd641d7ac0 con 0x7efd641430c0 2024-03-25T17:48:35.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.351+0000 7efd59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:35.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.351+0000 7efd59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:48:35.352 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:48:35.352 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:48:35.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 10 librados: watch_flush enter 2024-03-25T17:48:35.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 10 librados: watch_flush exit 2024-03-25T17:48:35.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T17:48:35.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 10 client.4639.objecter successfully canceled tick 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 >> v1:172.21.15.44:6800/3257043132 conn(0x7efd4806df00 legacy=0x7efd480703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 10 monclient: shutdown 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 >> v1:172.21.15.111:6789/0 conn(0x7efd641430c0 legacy=0x7efd64138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 shutdown_connections 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 >> 172.21.15.44:0/488541602 conn(0x7efd640b7ea0 msgr2=0x7efd64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 shutdown_connections 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.353+0000 7efd6b9cf640 1 -- 172.21.15.44:0/488541602 wait complete. 2024-03-25T17:48:35.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:35.354+0000 7efd6b9cf640 1 librados: shutdown 2024-03-25T17:48:35.366 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:48:39.092 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:56.447 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:39.139 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:56.494 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:39.174 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:56.529 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:39.185 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:56.505 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:39.198 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:56.554 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:39.200 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:56.520 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:39.209 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:56.565 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:39.223 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:56.543 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:39.234 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:56.590 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:42.173 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:48:42.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.288+0000 7f350db48640 10 monclient: get_monmap_and_config 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.288+0000 7f350db48640 10 monclient: build_initial_monmap 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.288+0000 7f350db48640 10 monclient: monmap: 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:48:42.289296+0000 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:48:42.289296+0000 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:48:42.289 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.289+0000 7f350db48640 1 Processor -- start 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.289+0000 7f350db48640 1 -- start start 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.289+0000 7f350db48640 10 monclient: init 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 10 monclient(hunting): picked mon.noname-c con 0x7f35080b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 10 monclient(hunting): picked mon.noname-a con 0x7f35080b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 10 monclient(hunting): picked mon.noname-b con 0x7f350814be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35080af940 con 0x7f35080b1730 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35080b0a70 con 0x7f35080b4d60 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35080b0d50 con 0x7f350814be90 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:42.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.290+0000 7f350db48640 10 monclient(hunting): authenticate will time out at 6525.517954s 2024-03-25T17:48:42.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.297+0000 7f3507fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f350814be90 0x7f350814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:59802/0 (socket says 172.21.15.44:59802) 2024-03-25T17:48:42.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.297+0000 7f3507fff640 1 -- 172.21.15.44:0/1329378319 learned_addr learned my addr 172.21.15.44:0/1329378319 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:42.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.312+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071479165 0 0) 0x7f35080af940 con 0x7f35080b1730 2024-03-25T17:48:42.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.312+0000 7f35067fc640 10 monclient(hunting): _negotiate 2024-03-25T17:48:42.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.312+0000 7f35067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:42.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.312+0000 7f35067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:42.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.312+0000 7f35067fc640 10 monclient(hunting): my global_id is 4650 2024-03-25T17:48:42.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.312+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f0003520 con 0x7f35080b1730 2024-03-25T17:48:42.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.313+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097484493 0 0) 0x7f35080b0d50 con 0x7f350814be90 2024-03-25T17:48:42.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.313+0000 7f35067fc640 10 monclient(hunting): _negotiate 2024-03-25T17:48:42.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.313+0000 7f35067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:42.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.313+0000 7f35067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:42.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.313+0000 7f35067fc640 10 monclient(hunting): my global_id is 4646 2024-03-25T17:48:42.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.313+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35080af940 con 0x7f350814be90 2024-03-25T17:48:42.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 781484980 0 0) 0x7f34f0003520 con 0x7f35080b1730 2024-03-25T17:48:42.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 >> v1:172.21.15.44:6790/0 conn(0x7f350814be90 legacy=0x7f350814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:42.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 >> v1:172.21.15.111:6789/0 conn(0x7f35080b4d60 legacy=0x7f350813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:42.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 10 monclient: _finish_hunting 0 2024-03-25T17:48:42.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 1 monclient: found mon.noname-a 2024-03-25T17:48:42.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:42.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:48:42.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35080b1030 con 0x7f35080b1730 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 10 monclient: _finish_auth 0 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 10 monclient: _check_auth_tickets 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f34fc001690 con 0x7f35080b1730 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f350db48640 5 monclient: authenticate success, global_id 4650 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f350db48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f35067fc640 10 monclient: dump: 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:42.321 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:42.322 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:42.322 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:42.322 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:42.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.319+0000 7f350db48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:42.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34fc0017f0 con 0x7f35080b1730 2024-03-25T17:48:42.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f35067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:42.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f35067fc640 1 -- 172.21.15.44:0/1329378319 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f34fc001c00 con 0x7f35080b1730 2024-03-25T17:48:42.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f350db48640 10 monclient: get_monmap_and_config success 2024-03-25T17:48:42.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f350db48640 4 set_mon_vals no callback set 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f35067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f35067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f35067fc640 10 monclient: dump: 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:42.324 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:42.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f350db48640 10 monclient: shutdown 2024-03-25T17:48:42.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f350db48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f350db48640 1 -- 172.21.15.44:0/1329378319 >> v1:172.21.15.44:6789/0 conn(0x7f35080b1730 legacy=0x7f35080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.323+0000 7f350db48640 1 -- 172.21.15.44:0/1329378319 shutdown_connections 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.324+0000 7f350db48640 1 -- 172.21.15.44:0/1329378319 >> 172.21.15.44:0/1329378319 conn(0x7f35080617a0 msgr2=0x7f3508061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.324+0000 7f350db48640 1 -- 172.21.15.44:0/1329378319 shutdown_connections 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.324+0000 7f350db48640 1 -- 172.21.15.44:0/1329378319 wait complete. 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.324+0000 7f350db48640 10 monclient: build_initial_monmap 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.324+0000 7f350db48640 10 monclient: monmap: 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:42.326 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.325+0000 7f350db48640 1 librados: starting msgr at 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.325+0000 7f350db48640 1 librados: starting objecter 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.325+0000 7f350db48640 1 Processor -- start 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.325+0000 7f350db48640 1 -- start start 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.325+0000 7f350db48640 1 librados: setting wanted keys 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.325+0000 7f350db48640 1 librados: calling monclient init 2024-03-25T17:48:42.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.325+0000 7f350db48640 10 monclient: init 2024-03-25T17:48:42.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:42.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:48:42.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 10 monclient(hunting): picked mon.noname-a con 0x7f35080b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 10 monclient(hunting): picked mon.noname-b con 0x7f35080b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 10 monclient(hunting): picked mon.noname-c con 0x7f350814be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3508145910 con 0x7f35080b1730 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3508135810 con 0x7f35080b4d60 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35081453f0 con 0x7f350814be90 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.326+0000 7f350db48640 10 monclient(hunting): authenticate will time out at 6525.553891s 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.328+0000 7f3507fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f350814be90 0x7f3508135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:59830/0 (socket says 172.21.15.44:59830) 2024-03-25T17:48:42.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.328+0000 7f3507fff640 1 -- 172.21.15.44:0/909951004 learned_addr learned my addr 172.21.15.44:0/909951004 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:42.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.329+0000 7f34effff640 10 client.?.objecter ms_handle_connect 0x7f350814be90 2024-03-25T17:48:42.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.329+0000 7f34effff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:42.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.331+0000 7f34effff640 10 client.?.objecter ms_handle_connect 0x7f35080b1730 2024-03-25T17:48:42.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.331+0000 7f34effff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:42.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.336+0000 7f34effff640 10 client.?.objecter ms_handle_connect 0x7f35080b4d60 2024-03-25T17:48:42.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.336+0000 7f34effff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:42.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.337+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330153681 0 0) 0x7f3508145910 con 0x7f35080b1730 2024-03-25T17:48:42.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.337+0000 7f34effff640 10 monclient(hunting): _negotiate 2024-03-25T17:48:42.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.337+0000 7f34effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:42.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.337+0000 7f34effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:42.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.337+0000 7f34effff640 10 monclient(hunting): my global_id is 4653 2024-03-25T17:48:42.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.337+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f4004eb0 con 0x7f35080b1730 2024-03-25T17:48:42.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.346+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979404211 0 0) 0x7f3508135810 con 0x7f35080b4d60 2024-03-25T17:48:42.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.346+0000 7f34effff640 10 monclient(hunting): _negotiate 2024-03-25T17:48:42.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.346+0000 7f34effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:42.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.346+0000 7f34effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:42.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.346+0000 7f34effff640 10 monclient(hunting): my global_id is 4645 2024-03-25T17:48:42.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.346+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3508145910 con 0x7f35080b4d60 2024-03-25T17:48:42.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.347+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177266642 0 0) 0x7f35081453f0 con 0x7f350814be90 2024-03-25T17:48:42.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.347+0000 7f34effff640 10 monclient(hunting): _negotiate 2024-03-25T17:48:42.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.347+0000 7f34effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:42.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.347+0000 7f34effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:42.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.347+0000 7f34effff640 10 monclient(hunting): my global_id is 4649 2024-03-25T17:48:42.348 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.347+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3508135810 con 0x7f350814be90 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1054320068 0 0) 0x7f34f4004eb0 con 0x7f35080b1730 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35081453f0 con 0x7f35080b1730 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f34f80018c0 con 0x7f35080b1730 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 10 monclient(hunting): dump: 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.364 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730705657 0 0) 0x7f3508145910 con 0x7f35080b4d60 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34f4004eb0 con 0x7f35080b4d60 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f34fc001b40 con 0x7f35080b4d60 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.363+0000 7f34effff640 10 monclient(hunting): dump: 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:42.365 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.366 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.366 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:42.366 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:42.366 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:42.366 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:42.366 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:42.366 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:42.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1209134166 0 0) 0x7f34f4004eb0 con 0x7f35080b4d60 2024-03-25T17:48:42.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 >> v1:172.21.15.44:6790/0 conn(0x7f350814be90 legacy=0x7f3508135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:42.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 >> v1:172.21.15.44:6789/0 conn(0x7f35080b1730 legacy=0x7f35081d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:42.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 10 monclient: _finish_hunting 0 2024-03-25T17:48:42.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 1 monclient: found mon.b 2024-03-25T17:48:42.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:42.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508137bc0 con 0x7f35080b4d60 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 10 monclient: _finish_auth 0 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 10 monclient: _check_auth_tickets 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f34effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 5 monclient: authenticate success, global_id 4645 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 10 monclient: _renew_subs 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35081405f0 con 0x7f35080b4d60 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 10 monclient: _renew_subs 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35081d7da0 con 0x7f35080b4d60 2024-03-25T17:48:42.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.374+0000 7f350db48640 1 librados: init done 2024-03-25T17:48:42.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.375+0000 7f350db48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:42.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.375+0000 7f350db48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:42.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.375+0000 7f350db48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:42.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.375+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3508145580 con 0x7f35080b4d60 2024-03-25T17:48:42.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.645+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34fc0038a0 con 0x7f35080b4d60 2024-03-25T17:48:42.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.645+0000 7f34effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.645+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f34fc003bb0 con 0x7f35080b4d60 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.645+0000 7f34effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.645+0000 7f34effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.645+0000 7f34effff640 10 monclient: dump: 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:42.646 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:42.647 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.645+0000 7f3505ffb640 4 set_mon_vals no callback set 2024-03-25T17:48:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.645+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f34fc019b80 con 0x7f35080b4d60 2024-03-25T17:48:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.646+0000 7f35077fe640 10 monclient: get_auth_request con 0x7f34f406e040 auth_method 0 2024-03-25T17:48:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.646+0000 7f34effff640 10 client.4645.objecter ms_handle_connect 0x7f34f406e040 2024-03-25T17:48:42.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.650+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f34fc0536a0 con 0x7f35080b4d60 2024-03-25T17:48:42.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.650+0000 7f34effff640 10 client.4645.objecter ms_dispatch 0x7f3508000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:48:42.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.650+0000 7f34effff640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:48:42.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.650+0000 7f34effff640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:48:42.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.650+0000 7f34effff640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T17:48:42.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.650+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34fc04f1b0 con 0x7f35080b4d60 2024-03-25T17:48:42.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.650+0000 7f34effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:42.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.650+0000 7f34effff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:48:42.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.907+0000 7f350db48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:42.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.907+0000 7f350db48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:42.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.907+0000 7f350db48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:42.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.907+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35080627b0 con 0x7f35080b4d60 2024-03-25T17:48:42.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.928+0000 7f34effff640 1 -- 172.21.15.44:0/909951004 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f34fc0510e0 con 0x7f35080b4d60 2024-03-25T17:48:42.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.928+0000 7f34effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:42.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.928+0000 7f34effff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:48:42.929 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:48:42.929 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 10 librados: watch_flush enter 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 10 librados: watch_flush exit 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 10 client.4645.objecter successfully canceled tick 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 >> v1:172.21.15.44:6800/3257043132 conn(0x7f34f406e040 legacy=0x7f34f4070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 10 monclient: shutdown 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 >> v1:172.21.15.111:6789/0 conn(0x7f35080b4d60 legacy=0x7f35081349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 shutdown_connections 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 >> 172.21.15.44:0/909951004 conn(0x7f35080617a0 msgr2=0x7f350813c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 shutdown_connections 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 1 -- 172.21.15.44:0/909951004 wait complete. 2024-03-25T17:48:42.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:42.931+0000 7f350db48640 1 librados: shutdown 2024-03-25T17:48:42.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:48:49.749 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:48:49.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.866+0000 7fd12e790640 10 monclient: get_monmap_and_config 2024-03-25T17:48:49.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.866+0000 7fd12e790640 10 monclient: build_initial_monmap 2024-03-25T17:48:49.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.866+0000 7fd12e790640 10 monclient: monmap: 2024-03-25T17:48:49.867 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:49.867 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:49.867 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:48:49.867679+0000 2024-03-25T17:48:49.868 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:48:49.867679+0000 2024-03-25T17:48:49.868 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:49.868 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:49.868 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:49.868 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:48:49.868 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:48:49.868 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:49.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.867+0000 7fd12e790640 1 Processor -- start 2024-03-25T17:48:49.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.867+0000 7fd12e790640 1 -- start start 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.867+0000 7fd12e790640 10 monclient: init 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.868+0000 7fd12e790640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.868+0000 7fd12e790640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.868+0000 7fd12e790640 10 monclient(hunting): picked mon.noname-b con 0x7fd1280a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.868+0000 7fd12e790640 10 monclient(hunting): picked mon.noname-c con 0x7fd12813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.868+0000 7fd12e790640 10 monclient(hunting): picked mon.noname-a con 0x7fd1281430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.868+0000 7fd12e790640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd128142520 con 0x7fd1281430c0 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.868+0000 7fd12e790640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1280b28a0 con 0x7fd12813f710 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.869+0000 7fd12e790640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1280b2a30 con 0x7fd1280a98f0 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.869+0000 7fd12e790640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:49.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.869+0000 7fd12e790640 10 monclient(hunting): authenticate will time out at 6533.096828s 2024-03-25T17:48:49.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.876+0000 7fd1277fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd12813f710 0x7fd1281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:53178/0 (socket says 172.21.15.44:53178) 2024-03-25T17:48:49.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.876+0000 7fd1277fe640 1 -- 172.21.15.44:0/2373776414 learned_addr learned my addr 172.21.15.44:0/2373776414 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:49.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.887+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914395653 0 0) 0x7fd128142520 con 0x7fd1281430c0 2024-03-25T17:48:49.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.887+0000 7fd126ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:49.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.887+0000 7fd126ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:49.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.887+0000 7fd126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:49.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.887+0000 7fd126ffd640 10 monclient(hunting): my global_id is 4656 2024-03-25T17:48:49.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.887+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd110003520 con 0x7fd1281430c0 2024-03-25T17:48:49.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.889+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41878794 0 0) 0x7fd1280b28a0 con 0x7fd12813f710 2024-03-25T17:48:49.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.889+0000 7fd126ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:49.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.889+0000 7fd126ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:49.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.889+0000 7fd126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:49.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.889+0000 7fd126ffd640 10 monclient(hunting): my global_id is 4648 2024-03-25T17:48:49.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.889+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd128142520 con 0x7fd12813f710 2024-03-25T17:48:49.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3485706276 0 0) 0x7fd128142520 con 0x7fd12813f710 2024-03-25T17:48:49.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 >> v1:172.21.15.44:6790/0 conn(0x7fd1280a98f0 legacy=0x7fd1280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:49.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 >> v1:172.21.15.44:6789/0 conn(0x7fd1281430c0 legacy=0x7fd12814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 1 monclient: found mon.noname-c 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1280b44c0 con 0x7fd12813f710 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 10 monclient: _finish_auth 0 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd11c001670 con 0x7fd12813f710 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd12e790640 5 monclient: authenticate success, global_id 4648 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd12e790640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd126ffd640 10 monclient: dump: 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:49.899 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:49.900 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.898+0000 7fd12e790640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:49.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd11c0017d0 con 0x7fd12813f710 2024-03-25T17:48:49.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd126ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:49.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd126ffd640 1 -- 172.21.15.44:0/2373776414 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd11c001ba0 con 0x7fd12813f710 2024-03-25T17:48:49.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 10 monclient: get_monmap_and_config success 2024-03-25T17:48:49.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 4 set_mon_vals no callback set 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd126ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd126ffd640 10 monclient: dump: 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:49.909 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:49.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 10 monclient: shutdown 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 1 -- 172.21.15.44:0/2373776414 >> v1:172.21.15.111:6789/0 conn(0x7fd12813f710 legacy=0x7fd1281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 1 -- 172.21.15.44:0/2373776414 shutdown_connections 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 1 -- 172.21.15.44:0/2373776414 >> 172.21.15.44:0/2373776414 conn(0x7fd1280b7ea0 msgr2=0x7fd1280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 1 -- 172.21.15.44:0/2373776414 shutdown_connections 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.908+0000 7fd12e790640 1 -- 172.21.15.44:0/2373776414 wait complete. 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.909+0000 7fd12e790640 10 monclient: build_initial_monmap 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.909+0000 7fd12e790640 10 monclient: monmap: 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:49.912 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.909+0000 7fd12e790640 1 librados: starting msgr at 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.909+0000 7fd12e790640 1 librados: starting objecter 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 1 Processor -- start 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 1 -- start start 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 1 librados: setting wanted keys 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 1 librados: calling monclient init 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 10 monclient: init 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 10 monclient(hunting): picked mon.noname-a con 0x7fd1280a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 10 monclient(hunting): picked mon.noname-c con 0x7fd12813f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 10 monclient(hunting): picked mon.noname-b con 0x7fd1281430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:49.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1281db660 con 0x7fd1280a98f0 2024-03-25T17:48:49.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.910+0000 7fd12e790640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1281db7f0 con 0x7fd1281430c0 2024-03-25T17:48:49.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.911+0000 7fd12e790640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12814dee0 con 0x7fd12813f710 2024-03-25T17:48:49.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.911+0000 7fd12e790640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:49.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.911+0000 7fd12e790640 10 monclient(hunting): authenticate will time out at 6533.138707s 2024-03-25T17:48:49.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.924+0000 7fd1277fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fd12813f710 0x7fd1280ae760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:47300/0 (socket says 172.21.15.44:47300) 2024-03-25T17:48:49.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.924+0000 7fd1277fe640 1 -- 172.21.15.44:0/3405498157 learned_addr learned my addr 172.21.15.44:0/3405498157 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:49.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.927+0000 7fd124ff9640 10 client.?.objecter ms_handle_connect 0x7fd12813f710 2024-03-25T17:48:49.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.927+0000 7fd124ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:49.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.934+0000 7fd124ff9640 10 client.?.objecter ms_handle_connect 0x7fd1281430c0 2024-03-25T17:48:49.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.934+0000 7fd124ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:49.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.939+0000 7fd124ff9640 10 client.?.objecter ms_handle_connect 0x7fd1280a98f0 2024-03-25T17:48:49.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.939+0000 7fd124ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:49.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.943+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373956343 0 0) 0x7fd12814dee0 con 0x7fd12813f710 2024-03-25T17:48:49.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.943+0000 7fd124ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:48:49.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.943+0000 7fd124ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:49.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.943+0000 7fd124ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:49.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.943+0000 7fd124ff9640 10 monclient(hunting): my global_id is 4655 2024-03-25T17:48:49.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.943+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd114004eb0 con 0x7fd12813f710 2024-03-25T17:48:49.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.953+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765642014 0 0) 0x7fd1281db7f0 con 0x7fd1281430c0 2024-03-25T17:48:49.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.953+0000 7fd124ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:48:49.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.953+0000 7fd124ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:49.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.953+0000 7fd124ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:49.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.953+0000 7fd124ff9640 10 monclient(hunting): my global_id is 4651 2024-03-25T17:48:49.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.953+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd12814dee0 con 0x7fd1281430c0 2024-03-25T17:48:49.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.955+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204307245 0 0) 0x7fd1281db660 con 0x7fd1280a98f0 2024-03-25T17:48:49.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.955+0000 7fd124ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:48:49.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.955+0000 7fd124ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:49.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.955+0000 7fd124ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:49.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.955+0000 7fd124ff9640 10 monclient(hunting): my global_id is 4659 2024-03-25T17:48:49.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.956+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1281db7f0 con 0x7fd1280a98f0 2024-03-25T17:48:49.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.966+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 562285956 0 0) 0x7fd114004eb0 con 0x7fd12813f710 2024-03-25T17:48:49.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.966+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1281db660 con 0x7fd12813f710 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.966+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd11c001ba0 con 0x7fd12813f710 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.966+0000 7fd124ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.966+0000 7fd124ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.966+0000 7fd124ff9640 10 monclient(hunting): dump: 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:49.967 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:49.968 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:49.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.966+0000 7fd124ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:48:49.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.982+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 422327266 0 0) 0x7fd12814dee0 con 0x7fd1281430c0 2024-03-25T17:48:49.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.982+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd114004eb0 con 0x7fd1281430c0 2024-03-25T17:48:49.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.982+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd120001a70 con 0x7fd1281430c0 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.982+0000 7fd124ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.982+0000 7fd124ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.982+0000 7fd124ff9640 10 monclient(hunting): dump: 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:49.983 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2212281671 0 0) 0x7fd114004eb0 con 0x7fd1281430c0 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 >> v1:172.21.15.44:6790/0 conn(0x7fd12813f710 legacy=0x7fd1280ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 >> v1:172.21.15.44:6789/0 conn(0x7fd1280a98f0 legacy=0x7fd1281dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 1 monclient: found mon.b 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12814e070 con 0x7fd1281430c0 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 10 monclient: _finish_auth 0 2024-03-25T17:48:49.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd124ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 5 monclient: authenticate success, global_id 4651 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 10 monclient: _renew_subs 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1280b11e0 con 0x7fd1281430c0 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 10 monclient: _renew_subs 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1280b1710 con 0x7fd1281430c0 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.988+0000 7fd12e790640 1 librados: init done 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.989+0000 7fd12e790640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.989+0000 7fd12e790640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.989+0000 7fd12e790640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:49.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:49.989+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12814e260 con 0x7fd1281430c0 2024-03-25T17:48:50.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.300+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd120001f60 con 0x7fd1281430c0 2024-03-25T17:48:50.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.300+0000 7fd124ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:50.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.300+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fd120003b10 con 0x7fd1281430c0 2024-03-25T17:48:50.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.300+0000 7fd124ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:50.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.300+0000 7fd124ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:48:50.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.300+0000 7fd124ff9640 10 monclient: dump: 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:50.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.300+0000 7fd1267fc640 4 set_mon_vals no callback set 2024-03-25T17:48:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.301+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fd120003cf0 con 0x7fd1281430c0 2024-03-25T17:48:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.301+0000 7fd127fff640 10 monclient: get_auth_request con 0x7fd11406dfb0 auth_method 0 2024-03-25T17:48:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.302+0000 7fd124ff9640 10 client.4651.objecter ms_handle_connect 0x7fd11406dfb0 2024-03-25T17:48:50.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.306+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7fd1200537e0 con 0x7fd1281430c0 2024-03-25T17:48:50.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.306+0000 7fd124ff9640 10 client.4651.objecter ms_dispatch 0x7fd128000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:48:50.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.306+0000 7fd124ff9640 3 client.4651.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:48:50.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.306+0000 7fd124ff9640 3 client.4651.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:48:50.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.306+0000 7fd124ff9640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T17:48:50.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.307+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd12004f2f0 con 0x7fd1281430c0 2024-03-25T17:48:50.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.307+0000 7fd124ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:50.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.307+0000 7fd124ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:48:50.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.545+0000 7fd12e790640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:50.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.545+0000 7fd12e790640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:50.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.545+0000 7fd12e790640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:48:50.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.545+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1280627b0 con 0x7fd1281430c0 2024-03-25T17:48:50.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.631+0000 7fd124ff9640 1 -- 172.21.15.44:0/3405498157 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7fd120051220 con 0x7fd1281430c0 2024-03-25T17:48:50.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.631+0000 7fd124ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:50.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.631+0000 7fd124ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:48:50.633 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:48:50.633 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:48:50.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.633+0000 7fd12e790640 10 librados: watch_flush enter 2024-03-25T17:48:50.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 10 librados: watch_flush exit 2024-03-25T17:48:50.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T17:48:50.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 10 client.4651.objecter successfully canceled tick 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 >> v1:172.21.15.44:6800/3257043132 conn(0x7fd11406dfb0 legacy=0x7fd114070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 10 monclient: shutdown 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 >> v1:172.21.15.111:6789/0 conn(0x7fd1281430c0 legacy=0x7fd1281e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 shutdown_connections 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 >> 172.21.15.44:0/3405498157 conn(0x7fd1280b7ea0 msgr2=0x7fd128074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 shutdown_connections 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 1 -- 172.21.15.44:0/3405498157 wait complete. 2024-03-25T17:48:50.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:50.634+0000 7fd12e790640 1 librados: shutdown 2024-03-25T17:48:50.647 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:48:57.454 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:48:57.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.573+0000 7f05723cd640 10 monclient: get_monmap_and_config 2024-03-25T17:48:57.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.573+0000 7f05723cd640 10 monclient: build_initial_monmap 2024-03-25T17:48:57.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.573+0000 7f05723cd640 10 monclient: monmap: 2024-03-25T17:48:57.574 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:48:57.574737+0000 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:48:57.574737+0000 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:48:57.575 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.574+0000 7f05723cd640 1 Processor -- start 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.574+0000 7f05723cd640 1 -- start start 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.574+0000 7f05723cd640 10 monclient: init 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 10 monclient(hunting): picked mon.noname-c con 0x7f056c0b4f40 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 10 monclient(hunting): picked mon.noname-b con 0x7f056c13fd90 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 10 monclient(hunting): picked mon.noname-a con 0x7f056c0b4540 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0aa0c0 con 0x7f056c0b4540 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0ab170 con 0x7f056c0b4f40 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0a9ba0 con 0x7f056c13fd90 2024-03-25T17:48:57.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:57.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.575+0000 7f05723cd640 10 monclient(hunting): authenticate will time out at 6540.803233s 2024-03-25T17:48:57.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.584+0000 7f056b7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f056c13fd90 0x7f056c1423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:51622/0 (socket says 172.21.15.44:51622) 2024-03-25T17:48:57.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.584+0000 7f056b7fe640 1 -- 172.21.15.44:0/2298758763 learned_addr learned my addr 172.21.15.44:0/2298758763 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:57.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.597+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591550390 0 0) 0x7f056c0a9ba0 con 0x7f056c13fd90 2024-03-25T17:48:57.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.597+0000 7f056affd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:57.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.597+0000 7f056affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:57.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.597+0000 7f056affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:57.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.597+0000 7f056affd640 10 monclient(hunting): my global_id is 4658 2024-03-25T17:48:57.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.597+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0554003520 con 0x7f056c13fd90 2024-03-25T17:48:57.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.604+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485348171 0 0) 0x7f056c0ab170 con 0x7f056c0b4f40 2024-03-25T17:48:57.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.604+0000 7f056affd640 10 monclient(hunting): _negotiate 2024-03-25T17:48:57.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.604+0000 7f056affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:57.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.604+0000 7f056affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:57.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.604+0000 7f056affd640 10 monclient(hunting): my global_id is 4654 2024-03-25T17:48:57.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.604+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c0a9ba0 con 0x7f056c0b4f40 2024-03-25T17:48:57.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180702945 0 0) 0x7f0554003520 con 0x7f056c13fd90 2024-03-25T17:48:57.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 >> v1:172.21.15.111:6789/0 conn(0x7f056c0b4f40 legacy=0x7f056c13f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:57.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 >> v1:172.21.15.44:6789/0 conn(0x7f056c0b4540 legacy=0x7f056c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:57.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 10 monclient: _finish_hunting 0 2024-03-25T17:48:57.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 1 monclient: found mon.noname-b 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0a9680 con 0x7f056c13fd90 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 10 monclient: _finish_auth 0 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 10 monclient: _check_auth_tickets 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0560001690 con 0x7f056c13fd90 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f05723cd640 5 monclient: authenticate success, global_id 4658 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f05723cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f056affd640 10 monclient: dump: 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:57.614 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:57.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.612+0000 7f05723cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.619+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05600017f0 con 0x7f056c13fd90 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.619+0000 7f056affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.619+0000 7f056affd640 1 -- 172.21.15.44:0/2298758763 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0560001c00 con 0x7f056c13fd90 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.619+0000 7f05723cd640 10 monclient: get_monmap_and_config success 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.619+0000 7f05723cd640 4 set_mon_vals no callback set 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.619+0000 7f056affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.619+0000 7f056affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.619+0000 7f056affd640 10 monclient: dump: 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:57.620 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:57.621 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:57.621 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:57.621 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:57.621 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:57.621 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.620+0000 7f05723cd640 10 monclient: shutdown 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.620+0000 7f05723cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.620+0000 7f05723cd640 1 -- 172.21.15.44:0/2298758763 >> v1:172.21.15.44:6790/0 conn(0x7f056c13fd90 legacy=0x7f056c1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.620+0000 7f05723cd640 1 -- 172.21.15.44:0/2298758763 shutdown_connections 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.620+0000 7f05723cd640 1 -- 172.21.15.44:0/2298758763 >> 172.21.15.44:0/2298758763 conn(0x7f056c0617a0 msgr2=0x7f056c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.620+0000 7f05723cd640 1 -- 172.21.15.44:0/2298758763 shutdown_connections 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.620+0000 7f05723cd640 1 -- 172.21.15.44:0/2298758763 wait complete. 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 10 monclient: build_initial_monmap 2024-03-25T17:48:57.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 10 monclient: monmap: 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 1 librados: starting msgr at 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 1 librados: starting objecter 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 1 Processor -- start 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 1 -- start start 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 1 librados: setting wanted keys 2024-03-25T17:48:57.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 1 librados: calling monclient init 2024-03-25T17:48:57.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.621+0000 7f05723cd640 10 monclient: init 2024-03-25T17:48:57.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 10 monclient: _reopen_session rank -1 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 10 monclient(hunting): picked mon.noname-a con 0x7f056c0b4540 addr v1:172.21.15.44:6789/0 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 10 monclient(hunting): picked mon.noname-c con 0x7f056c0b4f40 addr v1:172.21.15.44:6790/0 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 10 monclient(hunting): picked mon.noname-b con 0x7f056c13fd90 addr v1:172.21.15.111:6789/0 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0b0120 con 0x7f056c0b4540 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0b11a0 con 0x7f056c13fd90 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0b1930 con 0x7f056c0b4f40 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 10 monclient(hunting): _renew_subs 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.622+0000 7f05723cd640 10 monclient(hunting): authenticate will time out at 6540.850174s 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.623+0000 7f056b7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f056c0b4f40 0x7f056c1da820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:51666/0 (socket says 172.21.15.44:51666) 2024-03-25T17:48:57.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.623+0000 7f056b7fe640 1 -- 172.21.15.44:0/2548626124 learned_addr learned my addr 172.21.15.44:0/2548626124 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:48:57.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.625+0000 7f0568ff9640 10 client.?.objecter ms_handle_connect 0x7f056c0b4f40 2024-03-25T17:48:57.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.625+0000 7f0568ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:57.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.631+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998919884 0 0) 0x7f056c0b1930 con 0x7f056c0b4f40 2024-03-25T17:48:57.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.631+0000 7f0568ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:48:57.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.631+0000 7f0568ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:48:57.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.631+0000 7f0568ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:48:57.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.631+0000 7f0568ff9640 10 monclient(hunting): my global_id is 4661 2024-03-25T17:48:57.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.631+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0558004eb0 con 0x7f056c0b4f40 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.638+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2982904794 0 0) 0x7f0558004eb0 con 0x7f056c0b4f40 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.638+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f056c0b1930 con 0x7f056c0b4f40 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.638+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0560001bc0 con 0x7f056c0b4f40 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.638+0000 7f0568ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.638+0000 7f0568ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.638+0000 7f0568ff9640 10 monclient(hunting): dump: 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:57.639 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:57.640 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:57.640 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:57.640 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:57.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.638+0000 7f0568ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:48:57.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.645+0000 7f0568ff9640 10 client.?.objecter ms_handle_connect 0x7f056c13fd90 2024-03-25T17:48:57.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.645+0000 7f0568ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1282788828 0 0) 0x7f056c0b1930 con 0x7f056c0b4f40 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 >> v1:172.21.15.111:6789/0 conn(0x7f056c13fd90 legacy=0x7f056c1daf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 >> v1:172.21.15.44:6789/0 conn(0x7f056c0b4540 legacy=0x7f056c138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 1 monclient: found mon.c 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c139900 con 0x7f056c0b4f40 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 10 monclient: _finish_auth 0 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f0568ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:48:57.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 5 monclient: authenticate success, global_id 4661 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 10 monclient: _renew_subs 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c139550 con 0x7f056c0b4f40 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 10 monclient: _renew_subs 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c0af000 con 0x7f056c0b4f40 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.652+0000 7f05723cd640 1 librados: init done 2024-03-25T17:48:57.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.653+0000 7f05723cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:57.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.653+0000 7f05723cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:57.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.653+0000 7f05723cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:57.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.653+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c0627b0 con 0x7f056c0b4f40 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.676+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0560001d20 con 0x7f056c0b4f40 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.676+0000 7f0568ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.676+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f0560004c70 con 0x7f056c0b4f40 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.676+0000 7f0568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.676+0000 7f0568ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.676+0000 7f0568ff9640 10 monclient: dump: 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:48:57.677 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:48:57.678 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:48:57.678 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:48:57.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.676+0000 7f056a7fc640 4 set_mon_vals no callback set 2024-03-25T17:48:57.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.677+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f056001ad00 con 0x7f056c0b4f40 2024-03-25T17:48:57.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.677+0000 7f056bfff640 10 monclient: get_auth_request con 0x7f055806dbb0 auth_method 0 2024-03-25T17:48:57.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.678+0000 7f0568ff9640 10 client.4661.objecter ms_handle_connect 0x7f055806dbb0 2024-03-25T17:48:57.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.976+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3195531167 0 0) 0x7f0560054970 con 0x7f056c0b4f40 2024-03-25T17:48:57.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.976+0000 7f0568ff9640 10 client.4661.objecter ms_dispatch 0x7f056c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:48:57.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.976+0000 7f0568ff9640 3 client.4661.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:48:57.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.976+0000 7f0568ff9640 3 client.4661.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:48:57.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.976+0000 7f0568ff9640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-25T17:48:57.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.976+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05600502b0 con 0x7f056c0b4f40 2024-03-25T17:48:57.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.976+0000 7f0568ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:48:57.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:57.976+0000 7f0568ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:48:58.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.210+0000 7f05723cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:58.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.210+0000 7f05723cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:58.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.210+0000 7f05723cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:48:58.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.210+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f056c069f40 con 0x7f056c0b4f40 2024-03-25T17:48:58.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.255+0000 7f0568ff9640 1 -- 172.21.15.44:0/2548626124 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 651219345) 0x7f05600523b0 con 0x7f056c0b4f40 2024-03-25T17:48:58.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.255+0000 7f0568ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:48:58.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.255+0000 7f0568ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:48:58.256 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:48:58.256 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:39:05.411440+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:05.411440+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 10 librados: watch_flush enter 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 10 librados: watch_flush exit 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 10 client.4661.objecter successfully canceled tick 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 >> v1:172.21.15.44:6800/3257043132 conn(0x7f055806dbb0 legacy=0x7f0558070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 10 monclient: shutdown 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 >> v1:172.21.15.44:6790/0 conn(0x7f056c0b4f40 legacy=0x7f056c1da820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 shutdown_connections 2024-03-25T17:48:58.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 >> 172.21.15.44:0/2548626124 conn(0x7f056c0617a0 msgr2=0x7f056c0b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:48:58.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 shutdown_connections 2024-03-25T17:48:58.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 1 -- 172.21.15.44:0/2548626124 wait complete. 2024-03-25T17:48:58.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:48:58.257+0000 7f05723cd640 1 librados: shutdown 2024-03-25T17:48:58.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:48:59.172 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:16.528 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:59.177 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:16.533 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:59.219 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:16.539 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:59.228 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:16.584 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:59.228 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:16.584 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:59.235 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:16.555 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:59.243 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:16.599 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:59.259 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:16.579 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:48:59.267 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:16.622 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:05.076 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:49:05.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.179+0000 7f72d596d640 10 monclient: get_monmap_and_config 2024-03-25T17:49:05.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.179+0000 7f72d596d640 10 monclient: build_initial_monmap 2024-03-25T17:49:05.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.180+0000 7f72d596d640 10 monclient: monmap: 2024-03-25T17:49:05.180 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:05.180 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:05.181 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:49:05.180920+0000 2024-03-25T17:49:05.181 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:49:05.180920+0000 2024-03-25T17:49:05.181 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:05.181 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:05.181 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:05.181 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:49:05.181 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:49:05.181 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:05.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.180+0000 7f72d596d640 1 Processor -- start 2024-03-25T17:49:05.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.180+0000 7f72d596d640 1 -- start start 2024-03-25T17:49:05.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.180+0000 7f72d596d640 10 monclient: init 2024-03-25T17:49:05.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:05.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:49:05.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 10 monclient(hunting): picked mon.noname-b con 0x7f72d00b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:05.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 10 monclient(hunting): picked mon.noname-a con 0x7f72d00b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:05.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 10 monclient(hunting): picked mon.noname-c con 0x7f72d014be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:05.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d00af940 con 0x7f72d00b1730 2024-03-25T17:49:05.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d00b0a70 con 0x7f72d014be90 2024-03-25T17:49:05.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d00b0d50 con 0x7f72d00b4d60 2024-03-25T17:49:05.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:05.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.181+0000 7f72d596d640 10 monclient(hunting): authenticate will time out at 6548.409117s 2024-03-25T17:49:05.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.199+0000 7f72ceffd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f72d00b4d60 0x7f72d013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:44548/0 (socket says 172.21.15.44:44548) 2024-03-25T17:49:05.200 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.199+0000 7f72ceffd640 1 -- 172.21.15.44:0/3988717126 learned_addr learned my addr 172.21.15.44:0/3988717126 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:05.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.219+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442944321 0 0) 0x7f72d00b0d50 con 0x7f72d00b4d60 2024-03-25T17:49:05.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.219+0000 7f72ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:49:05.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.219+0000 7f72ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:05.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.219+0000 7f72ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:05.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.219+0000 7f72ce7fc640 10 monclient(hunting): my global_id is 4664 2024-03-25T17:49:05.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.219+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72bc003520 con 0x7f72d00b4d60 2024-03-25T17:49:05.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.246+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649896446 0 0) 0x7f72d00b0a70 con 0x7f72d014be90 2024-03-25T17:49:05.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.246+0000 7f72ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:49:05.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.246+0000 7f72ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:05.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.246+0000 7f72ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:05.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.246+0000 7f72ce7fc640 10 monclient(hunting): my global_id is 4660 2024-03-25T17:49:05.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.246+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d00b0d50 con 0x7f72d014be90 2024-03-25T17:49:05.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 490177343 0 0) 0x7f72bc003520 con 0x7f72d00b4d60 2024-03-25T17:49:05.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 >> v1:172.21.15.111:6789/0 conn(0x7f72d014be90 legacy=0x7f72d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:05.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 >> v1:172.21.15.44:6789/0 conn(0x7f72d00b1730 legacy=0x7f72d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:05.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:49:05.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 1 monclient: found mon.noname-b 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d00b1030 con 0x7f72d00b4d60 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 10 monclient: _finish_auth 0 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f72c0001690 con 0x7f72d00b4d60 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72d596d640 5 monclient: authenticate success, global_id 4664 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72d596d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72ce7fc640 10 monclient: dump: 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:05.249 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:05.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.247+0000 7f72d596d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.257+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72c00017f0 con 0x7f72d00b4d60 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.257+0000 7f72ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.257+0000 7f72ce7fc640 1 -- 172.21.15.44:0/3988717126 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f72c0001c00 con 0x7f72d00b4d60 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.257+0000 7f72d596d640 10 monclient: get_monmap_and_config success 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.257+0000 7f72d596d640 4 set_mon_vals no callback set 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.257+0000 7f72ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.257+0000 7f72ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.257+0000 7f72ce7fc640 10 monclient: dump: 2024-03-25T17:49:05.258 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:05.259 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.258+0000 7f72d596d640 10 monclient: shutdown 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.258+0000 7f72d596d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.258+0000 7f72d596d640 1 -- 172.21.15.44:0/3988717126 >> v1:172.21.15.44:6790/0 conn(0x7f72d00b4d60 legacy=0x7f72d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.258+0000 7f72d596d640 1 -- 172.21.15.44:0/3988717126 shutdown_connections 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.258+0000 7f72d596d640 1 -- 172.21.15.44:0/3988717126 >> 172.21.15.44:0/3988717126 conn(0x7f72d00617a0 msgr2=0x7f72d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.258+0000 7f72d596d640 1 -- 172.21.15.44:0/3988717126 shutdown_connections 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.258+0000 7f72d596d640 1 -- 172.21.15.44:0/3988717126 wait complete. 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.259+0000 7f72d596d640 10 monclient: build_initial_monmap 2024-03-25T17:49:05.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.259+0000 7f72d596d640 10 monclient: monmap: 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.259+0000 7f72d596d640 1 librados: starting msgr at 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.259+0000 7f72d596d640 1 librados: starting objecter 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 1 Processor -- start 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 1 -- start start 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 1 librados: setting wanted keys 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 1 librados: calling monclient init 2024-03-25T17:49:05.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 10 monclient: init 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 10 monclient(hunting): picked mon.noname-c con 0x7f72d00b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 10 monclient(hunting): picked mon.noname-b con 0x7f72d00b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 10 monclient(hunting): picked mon.noname-a con 0x7f72d014be90 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d01361e0 con 0x7f72d014be90 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d0145080 con 0x7f72d00b4d60 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d0145210 con 0x7f72d00b1730 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:05.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.260+0000 7f72d596d640 10 monclient(hunting): authenticate will time out at 6548.488532s 2024-03-25T17:49:05.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.268+0000 7f72c7fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f72d00b4d60 0x7f72d01d76c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:48900/0 (socket says 172.21.15.44:48900) 2024-03-25T17:49:05.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.268+0000 7f72c7fff640 1 -- 172.21.15.44:0/2343407970 learned_addr learned my addr 172.21.15.44:0/2343407970 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:05.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.268+0000 7f72ccff9640 10 client.?.objecter ms_handle_connect 0x7f72d00b4d60 2024-03-25T17:49:05.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.268+0000 7f72ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:05.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.276+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604958216 0 0) 0x7f72d0145080 con 0x7f72d00b4d60 2024-03-25T17:49:05.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.276+0000 7f72ccff9640 10 monclient(hunting): _negotiate 2024-03-25T17:49:05.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.276+0000 7f72ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:05.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.276+0000 7f72ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:05.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.276+0000 7f72ccff9640 10 monclient(hunting): my global_id is 4663 2024-03-25T17:49:05.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.276+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72b0004eb0 con 0x7f72d00b4d60 2024-03-25T17:49:05.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.278+0000 7f72ccff9640 10 client.?.objecter ms_handle_connect 0x7f72d00b1730 2024-03-25T17:49:05.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.278+0000 7f72ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.284+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4177382111 0 0) 0x7f72b0004eb0 con 0x7f72d00b4d60 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.284+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72d0145080 con 0x7f72d00b4d60 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.284+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f72b80019d0 con 0x7f72d00b4d60 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.284+0000 7f72ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.284+0000 7f72ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.284+0000 7f72ccff9640 10 monclient(hunting): dump: 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:05.285 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:05.286 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:05.286 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:05.286 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:05.286 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:05.286 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:05.286 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:05.286 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:05.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.284+0000 7f72ccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1598916556 0 0) 0x7f72d0145080 con 0x7f72d00b4d60 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 >> v1:172.21.15.44:6790/0 conn(0x7f72d00b1730 legacy=0x7f72d012ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 >> v1:172.21.15.44:6789/0 conn(0x7f72d014be90 legacy=0x7f72d01e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 10 monclient: _finish_hunting 0 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 1 monclient: found mon.b 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d01306a0 con 0x7f72d00b4d60 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 10 monclient: _finish_auth 0 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 10 monclient: _check_auth_tickets 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:05.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 5 monclient: authenticate success, global_id 4663 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 10 monclient: _renew_subs 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d0145440 con 0x7f72d00b4d60 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 10 monclient: _renew_subs 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d0137ea0 con 0x7f72d00b4d60 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.294+0000 7f72d596d640 1 librados: init done 2024-03-25T17:49:05.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.295+0000 7f72d596d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:05.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.295+0000 7f72d596d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:05.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.295+0000 7f72d596d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:05.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.295+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d0130890 con 0x7f72d00b4d60 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.568+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72b8001c90 con 0x7f72d00b4d60 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.568+0000 7f72ccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.568+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f72b8003a60 con 0x7f72d00b4d60 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.568+0000 7f72ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.568+0000 7f72ccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.568+0000 7f72ccff9640 10 monclient: dump: 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:05.569 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:05.570 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:05.570 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:05.570 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:05.570 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:05.570 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:05.570 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:05.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.568+0000 7f72cdffb640 4 set_mon_vals no callback set 2024-03-25T17:49:05.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.569+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f72b8003c80 con 0x7f72d00b4d60 2024-03-25T17:49:05.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.569+0000 7f72ceffd640 10 monclient: get_auth_request con 0x7f72b006dc20 auth_method 0 2024-03-25T17:49:05.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.569+0000 7f72ccff9640 10 client.4663.objecter ms_handle_connect 0x7f72b006dc20 2024-03-25T17:49:05.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.573+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7f72b80548b0 con 0x7f72d00b4d60 2024-03-25T17:49:05.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.573+0000 7f72ccff9640 10 client.4663.objecter ms_dispatch 0x7f72d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:49:05.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.573+0000 7f72ccff9640 3 client.4663.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:49:05.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.573+0000 7f72ccff9640 3 client.4663.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:49:05.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.573+0000 7f72ccff9640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T17:49:05.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.573+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72b80503c0 con 0x7f72d00b4d60 2024-03-25T17:49:05.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.573+0000 7f72ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:05.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.573+0000 7f72ccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:49:05.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.828+0000 7f72d596d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:05.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.828+0000 7f72d596d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:05.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.828+0000 7f72d596d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:05.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.828+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72d00627b0 con 0x7f72d00b4d60 2024-03-25T17:49:05.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.867+0000 7f72ccff9640 1 -- 172.21.15.44:0/2343407970 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7f72b80522f0 con 0x7f72d00b4d60 2024-03-25T17:49:05.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.867+0000 7f72ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:05.868 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.867+0000 7f72ccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:49:05.868 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:49:05.868 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:49:05.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 10 librados: watch_flush enter 2024-03-25T17:49:05.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 10 librados: watch_flush exit 2024-03-25T17:49:05.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 10 client.4663.objecter successfully canceled tick 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 >> v1:172.21.15.44:6800/3257043132 conn(0x7f72b006dc20 legacy=0x7f72b00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 10 monclient: shutdown 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 >> v1:172.21.15.111:6789/0 conn(0x7f72d00b4d60 legacy=0x7f72d01d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 shutdown_connections 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.869+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 >> 172.21.15.44:0/2343407970 conn(0x7f72d00617a0 msgr2=0x7f72d00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.870+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 shutdown_connections 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.870+0000 7f72d596d640 1 -- 172.21.15.44:0/2343407970 wait complete. 2024-03-25T17:49:05.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:05.870+0000 7f72d596d640 1 librados: shutdown 2024-03-25T17:49:05.883 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:49:12.689 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.802+0000 7fe82c9f4640 10 monclient: get_monmap_and_config 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.802+0000 7fe82c9f4640 10 monclient: build_initial_monmap 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.802+0000 7fe82c9f4640 10 monclient: monmap: 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:49:12.803439+0000 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:49:12.803439+0000 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:12.803 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:12.804 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:49:12.804 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:49:12.804 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.803+0000 7fe82c9f4640 1 Processor -- start 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.803+0000 7fe82c9f4640 1 -- start start 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.803+0000 7fe82c9f4640 10 monclient: init 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 10 monclient(hunting): picked mon.noname-b con 0x7fe8280a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 10 monclient(hunting): picked mon.noname-c con 0x7fe82813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 10 monclient(hunting): picked mon.noname-a con 0x7fe8281430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe828142520 con 0x7fe8281430c0 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8280b28a0 con 0x7fe82813f710 2024-03-25T17:49:12.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8280b2a30 con 0x7fe8280a98f0 2024-03-25T17:49:12.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:12.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.804+0000 7fe82c9f4640 10 monclient(hunting): authenticate will time out at 6556.031803s 2024-03-25T17:49:12.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.809+0000 7fe82659c640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fe8280a98f0 0x7fe8280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:59864/0 (socket says 172.21.15.44:59864) 2024-03-25T17:49:12.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.809+0000 7fe82659c640 1 -- 172.21.15.44:0/71351624 learned_addr learned my addr 172.21.15.44:0/71351624 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:12.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.820+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351103175 0 0) 0x7fe8280b2a30 con 0x7fe8280a98f0 2024-03-25T17:49:12.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.820+0000 7fe825d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:49:12.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.820+0000 7fe825d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:12.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.820+0000 7fe825d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:12.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.820+0000 7fe825d9b640 10 monclient(hunting): my global_id is 4670 2024-03-25T17:49:12.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.820+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe814003520 con 0x7fe8280a98f0 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1021184358 0 0) 0x7fe814003520 con 0x7fe8280a98f0 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 >> v1:172.21.15.111:6789/0 conn(0x7fe82813f710 legacy=0x7fe8281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 >> v1:172.21.15.44:6789/0 conn(0x7fe8281430c0 legacy=0x7fe82814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 1 monclient: found mon.noname-b 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8280b44c0 con 0x7fe8280a98f0 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 10 monclient: _finish_auth 0 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:12.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe818001690 con 0x7fe8280a98f0 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe825d9b640 10 monclient: dump: 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe82c9f4640 5 monclient: authenticate success, global_id 4670 2024-03-25T17:49:12.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.832+0000 7fe82c9f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:12.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.847+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8180017f0 con 0x7fe8280a98f0 2024-03-25T17:49:12.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.847+0000 7fe825d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:12.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.847+0000 7fe825d9b640 1 -- 172.21.15.44:0/71351624 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe818001c00 con 0x7fe8280a98f0 2024-03-25T17:49:12.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.847+0000 7fe825d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:12.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.847+0000 7fe825d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:49:12.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.847+0000 7fe825d9b640 10 monclient: dump: 2024-03-25T17:49:12.848 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:12.848 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.847+0000 7fe82c9f4640 10 monclient: get_monmap_and_config success 2024-03-25T17:49:12.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.847+0000 7fe82c9f4640 4 set_mon_vals no callback set 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.848+0000 7fe82c9f4640 10 monclient: shutdown 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.848+0000 7fe82c9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.848+0000 7fe82c9f4640 1 -- 172.21.15.44:0/71351624 >> v1:172.21.15.44:6790/0 conn(0x7fe8280a98f0 legacy=0x7fe8280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.848+0000 7fe82c9f4640 1 -- 172.21.15.44:0/71351624 shutdown_connections 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.848+0000 7fe82c9f4640 1 -- 172.21.15.44:0/71351624 >> 172.21.15.44:0/71351624 conn(0x7fe8280b7ea0 msgr2=0x7fe8280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.849+0000 7fe82c9f4640 1 -- 172.21.15.44:0/71351624 shutdown_connections 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.849+0000 7fe82c9f4640 1 -- 172.21.15.44:0/71351624 wait complete. 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 10 monclient: build_initial_monmap 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 10 monclient: monmap: 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:49:12.851 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 1 librados: starting msgr at 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 1 librados: starting objecter 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 1 Processor -- start 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 1 -- start start 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 1 librados: setting wanted keys 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 1 librados: calling monclient init 2024-03-25T17:49:12.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.850+0000 7fe82c9f4640 10 monclient: init 2024-03-25T17:49:12.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:12.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:49:12.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 10 monclient(hunting): picked mon.noname-b con 0x7fe8280a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:12.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 10 monclient(hunting): picked mon.noname-a con 0x7fe82813f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:12.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 10 monclient(hunting): picked mon.noname-c con 0x7fe8281430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:12.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe828135730 con 0x7fe82813f710 2024-03-25T17:49:12.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8281358c0 con 0x7fe8280a98f0 2024-03-25T17:49:12.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe828135a50 con 0x7fe8281430c0 2024-03-25T17:49:12.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:12.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.851+0000 7fe82c9f4640 10 monclient(hunting): authenticate will time out at 6556.079262s 2024-03-25T17:49:12.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.857+0000 7fe82659c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe8280a98f0 0x7fe8281dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:38990/0 (socket says 172.21.15.44:38990) 2024-03-25T17:49:12.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.857+0000 7fe82659c640 1 -- 172.21.15.44:0/661482806 learned_addr learned my addr 172.21.15.44:0/661482806 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:12.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.859+0000 7fe81effd640 10 client.?.objecter ms_handle_connect 0x7fe8280a98f0 2024-03-25T17:49:12.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.859+0000 7fe81effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:12.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.865+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272743846 0 0) 0x7fe8281358c0 con 0x7fe8280a98f0 2024-03-25T17:49:12.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.865+0000 7fe81effd640 10 monclient(hunting): _negotiate 2024-03-25T17:49:12.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.865+0000 7fe81effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:12.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.865+0000 7fe81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:12.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.865+0000 7fe81effd640 10 monclient(hunting): my global_id is 4669 2024-03-25T17:49:12.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.865+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808004eb0 con 0x7fe8280a98f0 2024-03-25T17:49:12.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.867+0000 7fe81effd640 10 client.?.objecter ms_handle_connect 0x7fe8281430c0 2024-03-25T17:49:12.867 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.867+0000 7fe81effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:12.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.883+0000 7fe81effd640 10 client.?.objecter ms_handle_connect 0x7fe82813f710 2024-03-25T17:49:12.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.883+0000 7fe81effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:12.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.889+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2975021792 0 0) 0x7fe808004eb0 con 0x7fe8280a98f0 2024-03-25T17:49:12.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.889+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8281358c0 con 0x7fe8280a98f0 2024-03-25T17:49:12.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.890+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe818001c00 con 0x7fe8280a98f0 2024-03-25T17:49:12.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.890+0000 7fe81effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:12.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.890+0000 7fe81effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.890+0000 7fe81effd640 10 monclient(hunting): dump: 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:12.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.890+0000 7fe81effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:49:12.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.897+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224630568 0 0) 0x7fe828135a50 con 0x7fe8281430c0 2024-03-25T17:49:12.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.897+0000 7fe81effd640 10 monclient(hunting): _negotiate 2024-03-25T17:49:12.897 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.897+0000 7fe81effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:12.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.897+0000 7fe81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:12.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.897+0000 7fe81effd640 10 monclient(hunting): my global_id is 4673 2024-03-25T17:49:12.898 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.897+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808004eb0 con 0x7fe8281430c0 2024-03-25T17:49:12.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.904+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732481697 0 0) 0x7fe828135730 con 0x7fe82813f710 2024-03-25T17:49:12.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.904+0000 7fe81effd640 10 monclient(hunting): _negotiate 2024-03-25T17:49:12.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.904+0000 7fe81effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:12.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.904+0000 7fe81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:12.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.904+0000 7fe81effd640 10 monclient(hunting): my global_id is 4671 2024-03-25T17:49:12.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.904+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe828135a50 con 0x7fe82813f710 2024-03-25T17:49:12.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 750856313 0 0) 0x7fe8281358c0 con 0x7fe8280a98f0 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 >> v1:172.21.15.44:6790/0 conn(0x7fe8281430c0 legacy=0x7fe8281d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 >> v1:172.21.15.44:6789/0 conn(0x7fe82813f710 legacy=0x7fe8281d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 10 monclient: _finish_hunting 0 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 1 monclient: found mon.b 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8281d5000 con 0x7fe8280a98f0 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 10 monclient: _finish_auth 0 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 10 monclient: _check_auth_tickets 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe81effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 5 monclient: authenticate success, global_id 4669 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 10 monclient: _renew_subs 2024-03-25T17:49:12.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:12.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8281d52b0 con 0x7fe8280a98f0 2024-03-25T17:49:12.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:49:12.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 10 monclient: _renew_subs 2024-03-25T17:49:12.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:12.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82813c900 con 0x7fe8280a98f0 2024-03-25T17:49:12.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.909+0000 7fe82c9f4640 1 librados: init done 2024-03-25T17:49:12.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.911+0000 7fe82c9f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:12.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.911+0000 7fe82c9f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:12.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.911+0000 7fe82c9f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:12.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:12.911+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8281371c0 con 0x7fe8280a98f0 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.184+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe818004a90 con 0x7fe8280a98f0 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.184+0000 7fe81effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.184+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fe818004e60 con 0x7fe8280a98f0 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.184+0000 7fe81effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.184+0000 7fe81effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.184+0000 7fe81effd640 10 monclient: dump: 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:13.185 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:13.186 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:13.186 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:13.186 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:13.186 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:13.186 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:13.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.184+0000 7fe82559a640 4 set_mon_vals no callback set 2024-03-25T17:49:13.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.185+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fe818005080 con 0x7fe8280a98f0 2024-03-25T17:49:13.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.185+0000 7fe81ffff640 10 monclient: get_auth_request con 0x7fe80806de80 auth_method 0 2024-03-25T17:49:13.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.185+0000 7fe81effd640 10 client.4669.objecter ms_handle_connect 0x7fe80806de80 2024-03-25T17:49:13.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.190+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7fe8180549a0 con 0x7fe8280a98f0 2024-03-25T17:49:13.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.190+0000 7fe81effd640 10 client.4669.objecter ms_dispatch 0x7fe828000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:49:13.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.190+0000 7fe81effd640 3 client.4669.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:49:13.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.190+0000 7fe81effd640 3 client.4669.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:49:13.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.190+0000 7fe81effd640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T17:49:13.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.190+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8180504b0 con 0x7fe8280a98f0 2024-03-25T17:49:13.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.190+0000 7fe81effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:13.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.190+0000 7fe81effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:49:13.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.453+0000 7fe82c9f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:13.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.453+0000 7fe82c9f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:13.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.453+0000 7fe82c9f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:13.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.453+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8280627b0 con 0x7fe8280a98f0 2024-03-25T17:49:13.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.510+0000 7fe81effd640 1 -- 172.21.15.44:0/661482806 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7fe8180523e0 con 0x7fe8280a98f0 2024-03-25T17:49:13.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.510+0000 7fe81effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:13.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.510+0000 7fe81effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:49:13.511 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:49:13.511 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 10 librados: watch_flush enter 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 10 librados: watch_flush exit 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 10 client.4669.objecter successfully canceled tick 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 >> v1:172.21.15.44:6800/3257043132 conn(0x7fe80806de80 legacy=0x7fe808070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 10 monclient: shutdown 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 >> v1:172.21.15.111:6789/0 conn(0x7fe8280a98f0 legacy=0x7fe8281dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 shutdown_connections 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 >> 172.21.15.44:0/661482806 conn(0x7fe8280b7ea0 msgr2=0x7fe82814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:13.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 shutdown_connections 2024-03-25T17:49:13.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 1 -- 172.21.15.44:0/661482806 wait complete. 2024-03-25T17:49:13.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:13.512+0000 7fe82c9f4640 1 librados: shutdown 2024-03-25T17:49:13.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:49:19.413 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:36.769 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:19.422 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:36.777 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:19.453 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:36.809 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:19.474 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:36.794 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:19.479 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:36.799 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:19.485 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:36.840 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:19.489 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:36.845 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:19.503 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:36.823 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:19.533 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:36.889 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:20.331 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:49:20.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.438+0000 7f9ca38fb640 10 monclient: get_monmap_and_config 2024-03-25T17:49:20.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.438+0000 7f9ca38fb640 10 monclient: build_initial_monmap 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.438+0000 7f9ca38fb640 10 monclient: monmap: 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:49:20.439193+0000 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:49:20.439193+0000 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:49:20.439 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:20.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.438+0000 7f9ca38fb640 1 Processor -- start 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 1 -- start start 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 10 monclient: init 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 10 monclient(hunting): picked mon.noname-a con 0x7f9c9c0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 10 monclient(hunting): picked mon.noname-c con 0x7f9c9c0b1730 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 10 monclient(hunting): picked mon.noname-b con 0x7f9c9c14be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c0af940 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c0b0a70 con 0x7f9c9c0b1730 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c0b0d50 con 0x7f9c9c14be90 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:20.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.439+0000 7f9ca38fb640 10 monclient(hunting): authenticate will time out at 6563.667339s 2024-03-25T17:49:20.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.454+0000 7f9ca0e96640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9c9c0b1730 0x7f9c9c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46668/0 (socket says 172.21.15.44:46668) 2024-03-25T17:49:20.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.454+0000 7f9ca0e96640 1 -- 172.21.15.44:0/3435408068 learned_addr learned my addr 172.21.15.44:0/3435408068 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:20.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.479+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158318886 0 0) 0x7f9c9c0b0a70 con 0x7f9c9c0b1730 2024-03-25T17:49:20.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.479+0000 7f9c93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:49:20.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.479+0000 7f9c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:20.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.479+0000 7f9c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:20.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.479+0000 7f9c93fff640 10 monclient(hunting): my global_id is 4672 2024-03-25T17:49:20.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.479+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c88003520 con 0x7f9c9c0b1730 2024-03-25T17:49:20.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.481+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564595554 0 0) 0x7f9c9c0b0d50 con 0x7f9c9c14be90 2024-03-25T17:49:20.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.481+0000 7f9c93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:49:20.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.481+0000 7f9c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:20.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.481+0000 7f9c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:20.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.481+0000 7f9c93fff640 10 monclient(hunting): my global_id is 4676 2024-03-25T17:49:20.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.481+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c9c0b0a70 con 0x7f9c9c14be90 2024-03-25T17:49:20.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.486+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937756914 0 0) 0x7f9c9c0af940 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.486+0000 7f9c93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:49:20.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.486+0000 7f9c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:20.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.486+0000 7f9c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:20.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.486+0000 7f9c93fff640 10 monclient(hunting): my global_id is 4674 2024-03-25T17:49:20.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.486+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c9c0b0d50 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2060129148 0 0) 0x7f9c9c0b0d50 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 >> v1:172.21.15.44:6790/0 conn(0x7f9c9c14be90 legacy=0x7f9c9c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 >> v1:172.21.15.111:6789/0 conn(0x7f9c9c0b1730 legacy=0x7f9c9c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 10 monclient: _finish_hunting 0 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 1 monclient: found mon.noname-a 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c0b1030 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 10 monclient: _finish_auth 0 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 10 monclient: _check_auth_tickets 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9c980015b0 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9ca38fb640 5 monclient: authenticate success, global_id 4674 2024-03-25T17:49:20.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9c93fff640 10 monclient: dump: 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:20.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.519+0000 7f9ca38fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.584+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c98001710 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.584+0000 7f9c93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.584+0000 7f9c93fff640 1 -- 172.21.15.44:0/3435408068 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9c98001b60 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.584+0000 7f9c93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.584+0000 7f9c93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.584+0000 7f9c93fff640 10 monclient: dump: 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:20.585 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:20.586 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:20.586 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:20.586 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:20.586 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:20.586 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:20.586 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:20.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.584+0000 7f9ca38fb640 10 monclient: get_monmap_and_config success 2024-03-25T17:49:20.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.584+0000 7f9ca38fb640 4 set_mon_vals no callback set 2024-03-25T17:49:20.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.585+0000 7f9ca38fb640 10 monclient: shutdown 2024-03-25T17:49:20.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.585+0000 7f9ca38fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:20.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.585+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3435408068 >> v1:172.21.15.44:6789/0 conn(0x7f9c9c0b4d60 legacy=0x7f9c9c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:20.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.585+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3435408068 shutdown_connections 2024-03-25T17:49:20.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.585+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3435408068 >> 172.21.15.44:0/3435408068 conn(0x7f9c9c0617a0 msgr2=0x7f9c9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:20.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.585+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3435408068 shutdown_connections 2024-03-25T17:49:20.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.585+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3435408068 wait complete. 2024-03-25T17:49:20.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 10 monclient: build_initial_monmap 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 10 monclient: monmap: 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 1 librados: starting msgr at 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 1 librados: starting objecter 2024-03-25T17:49:20.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 1 Processor -- start 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 1 -- start start 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 1 librados: setting wanted keys 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 1 librados: calling monclient init 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.586+0000 7f9ca38fb640 10 monclient: init 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 10 monclient(hunting): picked mon.noname-a con 0x7f9c9c0b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 10 monclient(hunting): picked mon.noname-c con 0x7f9c9c0b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 10 monclient(hunting): picked mon.noname-b con 0x7f9c9c14be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c140930 con 0x7f9c9c0b1730 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c141870 con 0x7f9c9c14be90 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c1327e0 con 0x7f9c9c0b4d60 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:20.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.587+0000 7f9ca38fb640 10 monclient(hunting): authenticate will time out at 6563.815055s 2024-03-25T17:49:20.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.600+0000 7f9ca1697640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f9c9c0b1730 0x7f9c9c1dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:43444/0 (socket says 172.21.15.44:43444) 2024-03-25T17:49:20.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.600+0000 7f9ca1697640 1 -- 172.21.15.44:0/3382789596 learned_addr learned my addr 172.21.15.44:0/3382789596 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:20.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.607+0000 7f9c91ffb640 10 client.?.objecter ms_handle_connect 0x7f9c9c0b1730 2024-03-25T17:49:20.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.607+0000 7f9c91ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:20.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.618+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867064905 0 0) 0x7f9c9c140930 con 0x7f9c9c0b1730 2024-03-25T17:49:20.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.618+0000 7f9c91ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:49:20.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.618+0000 7f9c91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:20.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.618+0000 7f9c91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:20.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.618+0000 7f9c91ffb640 10 monclient(hunting): my global_id is 4677 2024-03-25T17:49:20.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.618+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c80004eb0 con 0x7f9c9c0b1730 2024-03-25T17:49:20.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.635+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2364746988 0 0) 0x7f9c80004eb0 con 0x7f9c9c0b1730 2024-03-25T17:49:20.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.635+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c9c140930 con 0x7f9c9c0b1730 2024-03-25T17:49:20.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.635+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9c98001b20 con 0x7f9c9c0b1730 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.635+0000 7f9c91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.635+0000 7f9c91ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.635+0000 7f9c91ffb640 10 monclient(hunting): dump: 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:20.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.635+0000 7f9c91ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:49:20.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.638+0000 7f9c91ffb640 10 client.?.objecter ms_handle_connect 0x7f9c9c14be90 2024-03-25T17:49:20.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.638+0000 7f9c91ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:20.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2714058748 0 0) 0x7f9c9c140930 con 0x7f9c9c0b1730 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 >> v1:172.21.15.44:6790/0 conn(0x7f9c9c0b4d60 legacy=0x7f9c9c1319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 >> v1:172.21.15.111:6789/0 conn(0x7f9c9c14be90 legacy=0x7f9c9c1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 1 monclient: found mon.a 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c141ae0 con 0x7f9c9c0b1730 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 10 monclient: _finish_auth 0 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9c91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9ca38fb640 5 monclient: authenticate success, global_id 4677 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.645+0000 7f9ca38fb640 10 monclient: _renew_subs 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.646+0000 7f9ca38fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:20.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.646+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c137850 con 0x7f9c9c0b1730 2024-03-25T17:49:20.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.646+0000 7f9ca38fb640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:49:20.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.646+0000 7f9ca38fb640 10 monclient: _renew_subs 2024-03-25T17:49:20.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.646+0000 7f9ca38fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:20.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.646+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c140560 con 0x7f9c9c0b1730 2024-03-25T17:49:20.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.646+0000 7f9ca38fb640 1 librados: init done 2024-03-25T17:49:20.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.647+0000 7f9ca38fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:20.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.647+0000 7f9ca38fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:20.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.647+0000 7f9ca38fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:20.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.647+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c1401f0 con 0x7f9c9c0b1730 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.659+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c980038a0 con 0x7f9c9c0b1730 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.659+0000 7f9c91ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.659+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f9c98003cf0 con 0x7f9c9c0b1730 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.659+0000 7f9c91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.659+0000 7f9c91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.659+0000 7f9c91ffb640 10 monclient: dump: 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:20.661 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:20.662 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:20.662 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:20.662 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:20.662 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:20.662 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:20.662 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:20.662 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:20.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.659+0000 7f9c92ffd640 4 set_mon_vals no callback set 2024-03-25T17:49:20.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.968+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f9c98003e50 con 0x7f9c9c0b1730 2024-03-25T17:49:20.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.969+0000 7f9ca0e96640 10 monclient: get_auth_request con 0x7f9c8006dd60 auth_method 0 2024-03-25T17:49:20.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.969+0000 7f9c91ffb640 10 client.4677.objecter ms_handle_connect 0x7f9c8006dd60 2024-03-25T17:49:20.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.975+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7f9c980549a0 con 0x7f9c9c0b1730 2024-03-25T17:49:20.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.975+0000 7f9c91ffb640 10 client.4677.objecter ms_dispatch 0x7f9c9c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:49:20.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.975+0000 7f9c91ffb640 3 client.4677.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:49:20.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.975+0000 7f9c91ffb640 3 client.4677.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:49:20.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.975+0000 7f9c91ffb640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T17:49:20.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.975+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c980504b0 con 0x7f9c9c0b1730 2024-03-25T17:49:20.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.975+0000 7f9c91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:20.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:20.975+0000 7f9c91ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:49:21.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.234+0000 7f9ca38fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:21.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.234+0000 7f9ca38fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:21.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.234+0000 7f9ca38fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:21.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.234+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c9c0627b0 con 0x7f9c9c0b1730 2024-03-25T17:49:21.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.266+0000 7f9c91ffb640 1 -- 172.21.15.44:0/3382789596 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7f9c980523e0 con 0x7f9c9c0b1730 2024-03-25T17:49:21.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.266+0000 7f9c91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:21.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.266+0000 7f9c91ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:49:21.267 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:49:21.267 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:49:21.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 10 librados: watch_flush enter 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 10 librados: watch_flush exit 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 10 client.4677.objecter successfully canceled tick 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 >> v1:172.21.15.44:6800/3257043132 conn(0x7f9c8006dd60 legacy=0x7f9c80070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 10 monclient: shutdown 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 >> v1:172.21.15.44:6789/0 conn(0x7f9c9c0b1730 legacy=0x7f9c9c1dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 shutdown_connections 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 >> 172.21.15.44:0/3382789596 conn(0x7f9c9c0617a0 msgr2=0x7f9c9c0b6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 shutdown_connections 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 1 -- 172.21.15.44:0/3382789596 wait complete. 2024-03-25T17:49:21.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:21.268+0000 7f9ca38fb640 1 librados: shutdown 2024-03-25T17:49:21.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:49:28.087 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.202+0000 7f09b0274640 10 monclient: get_monmap_and_config 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.202+0000 7f09b0274640 10 monclient: build_initial_monmap 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.202+0000 7f09b0274640 10 monclient: monmap: 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:49:28.203008+0000 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:49:28.203008+0000 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:28.203 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:49:28.204 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:49:28.204 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:28.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.202+0000 7f09b0274640 1 Processor -- start 2024-03-25T17:49:28.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.202+0000 7f09b0274640 1 -- start start 2024-03-25T17:49:28.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.202+0000 7f09b0274640 10 monclient: init 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 10 monclient(hunting): picked mon.noname-c con 0x7f09a80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 10 monclient(hunting): picked mon.noname-a con 0x7f09a813f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 10 monclient(hunting): picked mon.noname-b con 0x7f09a81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a8142520 con 0x7f09a813f710 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a80b28a0 con 0x7f09a80a98f0 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a80b2a30 con 0x7f09a81430c0 2024-03-25T17:49:28.205 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:28.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.203+0000 7f09b0274640 10 monclient(hunting): authenticate will time out at 6571.431545s 2024-03-25T17:49:28.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.206+0000 7f09ae010640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f09a80a98f0 0x7f09a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:51412/0 (socket says 172.21.15.44:51412) 2024-03-25T17:49:28.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.206+0000 7f09ae010640 1 -- 172.21.15.44:0/2435632038 learned_addr learned my addr 172.21.15.44:0/2435632038 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:28.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.217+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044539190 0 0) 0x7f09a80b28a0 con 0x7f09a80a98f0 2024-03-25T17:49:28.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.217+0000 7f09ad00e640 10 monclient(hunting): _negotiate 2024-03-25T17:49:28.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.217+0000 7f09ad00e640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:28.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.217+0000 7f09ad00e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:28.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.217+0000 7f09ad00e640 10 monclient(hunting): my global_id is 4678 2024-03-25T17:49:28.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.217+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0994003520 con 0x7f09a80a98f0 2024-03-25T17:49:28.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264747787 0 0) 0x7f09a8142520 con 0x7f09a813f710 2024-03-25T17:49:28.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient(hunting): _negotiate 2024-03-25T17:49:28.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:28.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient(hunting): my global_id is 4680 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a80b28a0 con 0x7f09a813f710 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2234682291 0 0) 0x7f0994003520 con 0x7f09a80a98f0 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 >> v1:172.21.15.44:6790/0 conn(0x7f09a81430c0 legacy=0x7f09a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 >> v1:172.21.15.44:6789/0 conn(0x7f09a813f710 legacy=0x7f09a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient: _finish_hunting 0 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 1 monclient: found mon.noname-c 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a80b44c0 con 0x7f09a80a98f0 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient: _finish_auth 0 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient: _check_auth_tickets 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09b0274640 5 monclient: authenticate success, global_id 4678 2024-03-25T17:49:28.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09a4001690 con 0x7f09a80a98f0 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09ad00e640 10 monclient: dump: 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:28.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.226+0000 7f09b0274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.234+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09a40017f0 con 0x7f09a80a98f0 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.234+0000 7f09ad00e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.234+0000 7f09ad00e640 1 -- 172.21.15.44:0/2435632038 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09a4001c00 con 0x7f09a80a98f0 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.234+0000 7f09b0274640 10 monclient: get_monmap_and_config success 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.234+0000 7f09b0274640 4 set_mon_vals no callback set 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.234+0000 7f09ad00e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.234+0000 7f09ad00e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.234+0000 7f09ad00e640 10 monclient: dump: 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:28.235 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:28.236 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:28.236 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:28.236 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:28.236 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:28.236 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:28.236 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:28.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.235+0000 7f09b0274640 10 monclient: shutdown 2024-03-25T17:49:28.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.235+0000 7f09b0274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:28.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.235+0000 7f09b0274640 1 -- 172.21.15.44:0/2435632038 >> v1:172.21.15.111:6789/0 conn(0x7f09a80a98f0 legacy=0x7f09a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:28.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.235+0000 7f09b0274640 1 -- 172.21.15.44:0/2435632038 shutdown_connections 2024-03-25T17:49:28.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.235+0000 7f09b0274640 1 -- 172.21.15.44:0/2435632038 >> 172.21.15.44:0/2435632038 conn(0x7f09a80b7ea0 msgr2=0x7f09a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:28.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.235+0000 7f09b0274640 1 -- 172.21.15.44:0/2435632038 shutdown_connections 2024-03-25T17:49:28.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.235+0000 7f09b0274640 1 -- 172.21.15.44:0/2435632038 wait complete. 2024-03-25T17:49:28.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.236+0000 7f09b0274640 10 monclient: build_initial_monmap 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.236+0000 7f09b0274640 10 monclient: monmap: 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.236+0000 7f09b0274640 1 librados: starting msgr at 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.236+0000 7f09b0274640 1 librados: starting objecter 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.236+0000 7f09b0274640 1 Processor -- start 2024-03-25T17:49:28.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 1 -- start start 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 1 librados: setting wanted keys 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 1 librados: calling monclient init 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 10 monclient: init 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 10 monclient(hunting): picked mon.noname-a con 0x7f09a80a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 10 monclient(hunting): picked mon.noname-b con 0x7f09a813f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 10 monclient(hunting): picked mon.noname-c con 0x7f09a81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a81d7380 con 0x7f09a80a98f0 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a8139840 con 0x7f09a813f710 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a81d8200 con 0x7f09a81430c0 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:28.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.237+0000 7f09b0274640 10 monclient(hunting): authenticate will time out at 6571.465410s 2024-03-25T17:49:28.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.238+0000 7f09ae811640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f09a81430c0 0x7f09a81daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35152/0 (socket says 172.21.15.44:35152) 2024-03-25T17:49:28.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.238+0000 7f09ae811640 1 -- 172.21.15.44:0/2717602326 learned_addr learned my addr 172.21.15.44:0/2717602326 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:28.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.238+0000 7f099effd640 10 client.?.objecter ms_handle_connect 0x7f09a81430c0 2024-03-25T17:49:28.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.238+0000 7f099effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:28.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.244+0000 7f099effd640 10 client.?.objecter ms_handle_connect 0x7f09a813f710 2024-03-25T17:49:28.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.244+0000 7f099effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:28.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.308+0000 7f099effd640 10 client.?.objecter ms_handle_connect 0x7f09a80a98f0 2024-03-25T17:49:28.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.308+0000 7f099effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:28.318 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.318+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278398335 0 0) 0x7f09a81d7380 con 0x7f09a80a98f0 2024-03-25T17:49:28.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.318+0000 7f099effd640 10 monclient(hunting): _negotiate 2024-03-25T17:49:28.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.318+0000 7f099effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:28.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.318+0000 7f099effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:28.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.318+0000 7f099effd640 10 monclient(hunting): my global_id is 4683 2024-03-25T17:49:28.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.318+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f098c004eb0 con 0x7f09a80a98f0 2024-03-25T17:49:28.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.337+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4220727509 0 0) 0x7f098c004eb0 con 0x7f09a80a98f0 2024-03-25T17:49:28.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.337+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09a81d7380 con 0x7f09a80a98f0 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.337+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09a4001b00 con 0x7f09a80a98f0 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.337+0000 7f099effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.337+0000 7f099effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.337+0000 7f099effd640 10 monclient(hunting): dump: 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:28.338 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:28.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.337+0000 7f099effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:49:28.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 325243611 0 0) 0x7f09a81d7380 con 0x7f09a80a98f0 2024-03-25T17:49:28.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 >> v1:172.21.15.44:6790/0 conn(0x7f09a81430c0 legacy=0x7f09a81daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:28.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 >> v1:172.21.15.111:6789/0 conn(0x7f09a813f710 legacy=0x7f09a81da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:28.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 10 monclient: _finish_hunting 0 2024-03-25T17:49:28.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 1 monclient: found mon.a 2024-03-25T17:49:28.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:28.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:28.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a81d75d0 con 0x7f09a80a98f0 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 10 monclient: _finish_auth 0 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 10 monclient: _check_auth_tickets 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f099effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 5 monclient: authenticate success, global_id 4683 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 10 monclient: _renew_subs 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a81db6a0 con 0x7f09a80a98f0 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 10 monclient: _renew_subs 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a8137910 con 0x7f09a80a98f0 2024-03-25T17:49:28.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.351+0000 7f09b0274640 1 librados: init done 2024-03-25T17:49:28.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.353+0000 7f09b0274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:28.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.353+0000 7f09b0274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:28.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.353+0000 7f09b0274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:28.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.353+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a80627b0 con 0x7f09a80a98f0 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.620+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09a4001220 con 0x7f09a80a98f0 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.620+0000 7f099effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.620+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f09a4004c30 con 0x7f09a80a98f0 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.620+0000 7f099effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.620+0000 7f099effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.620+0000 7f099effd640 10 monclient: dump: 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:28.621 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:28.622 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:28.622 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:28.622 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:28.622 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:28.622 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:28.622 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:28.622 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:28.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.620+0000 7f09ac80d640 4 set_mon_vals no callback set 2024-03-25T17:49:28.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.621+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f09a401ac00 con 0x7f09a80a98f0 2024-03-25T17:49:28.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.621+0000 7f09ad80f640 10 monclient: get_auth_request con 0x7f098c06dd70 auth_method 0 2024-03-25T17:49:28.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.622+0000 7f099effd640 10 client.4683.objecter ms_handle_connect 0x7f098c06dd70 2024-03-25T17:49:28.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.626+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7f09a4054860 con 0x7f09a80a98f0 2024-03-25T17:49:28.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.626+0000 7f099effd640 10 client.4683.objecter ms_dispatch 0x7f09a8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:49:28.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.626+0000 7f099effd640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:49:28.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.626+0000 7f099effd640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:49:28.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.626+0000 7f099effd640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T17:49:28.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.626+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09a40501a0 con 0x7f09a80a98f0 2024-03-25T17:49:28.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.626+0000 7f099effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:28.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.626+0000 7f099effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:49:28.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.850+0000 7f09b0274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:28.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.850+0000 7f09b0274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:28.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.850+0000 7f09b0274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:28.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.850+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09a81d7760 con 0x7f09a80a98f0 2024-03-25T17:49:28.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.881+0000 7f099effd640 1 -- 172.21.15.44:0/2717602326 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7f09a40522a0 con 0x7f09a80a98f0 2024-03-25T17:49:28.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.881+0000 7f099effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:28.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.881+0000 7f099effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:49:28.882 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:49:28.882 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:49:28.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.883+0000 7f09b0274640 10 librados: watch_flush enter 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.883+0000 7f09b0274640 10 librados: watch_flush exit 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.883+0000 7f09b0274640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.883+0000 7f09b0274640 10 client.4683.objecter successfully canceled tick 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 >> v1:172.21.15.44:6800/3257043132 conn(0x7f098c06dd70 legacy=0x7f098c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 10 monclient: shutdown 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 >> v1:172.21.15.44:6789/0 conn(0x7f09a80a98f0 legacy=0x7f09a81d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 shutdown_connections 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 >> 172.21.15.44:0/2717602326 conn(0x7f09a80b7ea0 msgr2=0x7f09a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 shutdown_connections 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 1 -- 172.21.15.44:0/2717602326 wait complete. 2024-03-25T17:49:28.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:28.884+0000 7f09b0274640 1 librados: shutdown 2024-03-25T17:49:28.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:49:35.704 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:49:35.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.819+0000 7f7ee421e640 10 monclient: get_monmap_and_config 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.819+0000 7f7ee421e640 10 monclient: build_initial_monmap 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.819+0000 7f7ee421e640 10 monclient: monmap: 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:49:35.820144+0000 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:49:35.820144+0000 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:49:35.820 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:35.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.820+0000 7f7ee421e640 1 Processor -- start 2024-03-25T17:49:35.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.820+0000 7f7ee421e640 1 -- start start 2024-03-25T17:49:35.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.820+0000 7f7ee421e640 10 monclient: init 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 10 monclient(hunting): picked mon.noname-c con 0x7f7edc0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 10 monclient(hunting): picked mon.noname-b con 0x7f7edc13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 10 monclient(hunting): picked mon.noname-a con 0x7f7edc1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc142520 con 0x7f7edc1430c0 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc0b28a0 con 0x7f7edc0a98f0 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc0b2a30 con 0x7f7edc13f710 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:35.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.821+0000 7f7ee421e640 10 monclient(hunting): authenticate will time out at 6579.049143s 2024-03-25T17:49:35.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.822+0000 7f7ee17b9640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f7edc13f710 0x7f7edc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:47402/0 (socket says 172.21.15.44:47402) 2024-03-25T17:49:35.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.822+0000 7f7ee17b9640 1 -- 172.21.15.44:0/1188122064 learned_addr learned my addr 172.21.15.44:0/1188122064 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:35.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.835+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349030756 0 0) 0x7f7edc0b2a30 con 0x7f7edc13f710 2024-03-25T17:49:35.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.835+0000 7f7ee0fb8640 10 monclient(hunting): _negotiate 2024-03-25T17:49:35.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.835+0000 7f7ee0fb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:35.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.835+0000 7f7ee0fb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:35.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.835+0000 7f7ee0fb8640 10 monclient(hunting): my global_id is 4682 2024-03-25T17:49:35.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.835+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8003520 con 0x7f7edc13f710 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566496480 0 0) 0x7f7ec8003520 con 0x7f7edc13f710 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 >> v1:172.21.15.111:6789/0 conn(0x7f7edc0a98f0 legacy=0x7f7edc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 >> v1:172.21.15.44:6789/0 conn(0x7f7edc1430c0 legacy=0x7f7edc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 10 monclient: _finish_hunting 0 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 1 monclient: found mon.noname-b 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc0b44c0 con 0x7f7edc13f710 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 10 monclient: _finish_auth 0 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 10 monclient: _check_auth_tickets 2024-03-25T17:49:35.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7ecc001690 con 0x7f7edc13f710 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee0fb8640 10 monclient: dump: 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee421e640 5 monclient: authenticate success, global_id 4682 2024-03-25T17:49:35.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.849+0000 7f7ee421e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:35.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.881+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ecc0017f0 con 0x7f7edc13f710 2024-03-25T17:49:35.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.881+0000 7f7ee0fb8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:35.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.881+0000 7f7ee0fb8640 1 -- 172.21.15.44:0/1188122064 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7ecc001c40 con 0x7f7edc13f710 2024-03-25T17:49:35.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.881+0000 7f7ee421e640 10 monclient: get_monmap_and_config success 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.881+0000 7f7ee421e640 4 set_mon_vals no callback set 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.881+0000 7f7ee0fb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.881+0000 7f7ee0fb8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.881+0000 7f7ee0fb8640 10 monclient: dump: 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:35.882 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.882+0000 7f7ee421e640 10 monclient: shutdown 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.882+0000 7f7ee421e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.882+0000 7f7ee421e640 1 -- 172.21.15.44:0/1188122064 >> v1:172.21.15.44:6790/0 conn(0x7f7edc13f710 legacy=0x7f7edc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.882+0000 7f7ee421e640 1 -- 172.21.15.44:0/1188122064 shutdown_connections 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.882+0000 7f7ee421e640 1 -- 172.21.15.44:0/1188122064 >> 172.21.15.44:0/1188122064 conn(0x7f7edc0b7ea0 msgr2=0x7f7edc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.882+0000 7f7ee421e640 1 -- 172.21.15.44:0/1188122064 shutdown_connections 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.882+0000 7f7ee421e640 1 -- 172.21.15.44:0/1188122064 wait complete. 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 10 monclient: build_initial_monmap 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 10 monclient: monmap: 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:35.884 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 1 librados: starting msgr at 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 1 librados: starting objecter 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 1 Processor -- start 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 1 -- start start 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 1 librados: setting wanted keys 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 1 librados: calling monclient init 2024-03-25T17:49:35.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.883+0000 7f7ee421e640 10 monclient: init 2024-03-25T17:49:35.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:35.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:49:35.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 10 monclient(hunting): picked mon.noname-b con 0x7f7edc0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:35.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 10 monclient(hunting): picked mon.noname-a con 0x7f7edc13f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:35.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 10 monclient(hunting): picked mon.noname-c con 0x7f7edc1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:35.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc06f000 con 0x7f7edc13f710 2024-03-25T17:49:35.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc06ff30 con 0x7f7edc0a98f0 2024-03-25T17:49:35.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc06dea0 con 0x7f7edc1430c0 2024-03-25T17:49:35.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:35.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.884+0000 7f7ee421e640 10 monclient(hunting): authenticate will time out at 6579.112170s 2024-03-25T17:49:35.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.894+0000 7f7ee17b9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7edc13f710 0x7f7edc070930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:51600/0 (socket says 172.21.15.44:51600) 2024-03-25T17:49:35.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.894+0000 7f7ee17b9640 1 -- 172.21.15.44:0/901287447 learned_addr learned my addr 172.21.15.44:0/901287447 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:35.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.895+0000 7f7ed27fc640 10 client.?.objecter ms_handle_connect 0x7f7edc0a98f0 2024-03-25T17:49:35.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.895+0000 7f7ed27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:35.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.896+0000 7f7ed27fc640 10 client.?.objecter ms_handle_connect 0x7f7edc13f710 2024-03-25T17:49:35.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.896+0000 7f7ed27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:35.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.905+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661775035 0 0) 0x7f7edc06ff30 con 0x7f7edc0a98f0 2024-03-25T17:49:35.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.906+0000 7f7ed27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:49:35.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.906+0000 7f7ed27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:35.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.906+0000 7f7ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:35.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.906+0000 7f7ed27fc640 10 monclient(hunting): my global_id is 4684 2024-03-25T17:49:35.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.906+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec0004eb0 con 0x7f7edc0a98f0 2024-03-25T17:49:35.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.907+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112981051 0 0) 0x7f7edc06f000 con 0x7f7edc13f710 2024-03-25T17:49:35.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.907+0000 7f7ed27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:49:35.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.907+0000 7f7ed27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:35.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.907+0000 7f7ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:35.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.907+0000 7f7ed27fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T17:49:35.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.907+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7edc06ff30 con 0x7f7edc13f710 2024-03-25T17:49:35.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.910+0000 7f7ed27fc640 10 client.?.objecter ms_handle_connect 0x7f7edc1430c0 2024-03-25T17:49:35.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.910+0000 7f7ed27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:35.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.920+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878128168 0 0) 0x7f7edc06ff30 con 0x7f7edc13f710 2024-03-25T17:49:35.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.920+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7edc06f000 con 0x7f7edc13f710 2024-03-25T17:49:35.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.920+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7ecc001b00 con 0x7f7edc13f710 2024-03-25T17:49:35.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.920+0000 7f7ed27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:35.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.920+0000 7f7ed27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:49:35.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.920+0000 7f7ed27fc640 10 monclient(hunting): dump: 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:35.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.920+0000 7f7ed27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:49:35.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.932+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140669971 0 0) 0x7f7edc06dea0 con 0x7f7edc1430c0 2024-03-25T17:49:35.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.933+0000 7f7ed27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:49:35.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.933+0000 7f7ed27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:35.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.933+0000 7f7ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:35.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.933+0000 7f7ed27fc640 10 monclient(hunting): my global_id is 4685 2024-03-25T17:49:35.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.933+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7edc06ff30 con 0x7f7edc1430c0 2024-03-25T17:49:35.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.937+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3582012186 0 0) 0x7f7ec0004eb0 con 0x7f7edc0a98f0 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.937+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7edc06dea0 con 0x7f7edc0a98f0 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.937+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7ed8001930 con 0x7f7edc0a98f0 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.937+0000 7f7ed27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.937+0000 7f7ed27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.937+0000 7f7ed27fc640 10 monclient(hunting): dump: 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:35.938 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:35.939 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:35.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3089566615 0 0) 0x7f7edc06f000 con 0x7f7edc13f710 2024-03-25T17:49:35.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 >> v1:172.21.15.44:6790/0 conn(0x7f7edc1430c0 legacy=0x7f7edc071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:35.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 >> v1:172.21.15.111:6789/0 conn(0x7f7edc0a98f0 legacy=0x7f7edc06e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:35.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 10 monclient: _finish_hunting 0 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 1 monclient: found mon.a 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc139930 con 0x7f7edc13f710 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 10 monclient: _finish_auth 0 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 10 monclient: _check_auth_tickets 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ed27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 5 monclient: authenticate success, global_id 4689 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 10 monclient: _renew_subs 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc1e76b0 con 0x7f7edc13f710 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 10 monclient: _renew_subs 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc1e7a20 con 0x7f7edc13f710 2024-03-25T17:49:35.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.943+0000 7f7ee421e640 1 librados: init done 2024-03-25T17:49:35.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.944+0000 7f7ee421e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:35.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.944+0000 7f7ee421e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:35.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.944+0000 7f7ee421e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:35.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:35.944+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7edc0627b0 con 0x7f7edc13f710 2024-03-25T17:49:36.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.264+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ecc001ce0 con 0x7f7edc13f710 2024-03-25T17:49:36.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.264+0000 7f7ed27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.264+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7ecc004d90 con 0x7f7edc13f710 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.264+0000 7f7ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.264+0000 7f7ed27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.264+0000 7f7ed27fc640 10 monclient: dump: 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:36.265 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:36.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.264+0000 7f7ed3fff640 4 set_mon_vals no callback set 2024-03-25T17:49:36.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.265+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f7ecc01ad60 con 0x7f7edc13f710 2024-03-25T17:49:36.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.265+0000 7f7ee1fba640 10 monclient: get_auth_request con 0x7f7ec006df70 auth_method 0 2024-03-25T17:49:36.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.266+0000 7f7ed27fc640 10 client.4689.objecter ms_handle_connect 0x7f7ec006df70 2024-03-25T17:49:36.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.273+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7f7ecc0548f0 con 0x7f7edc13f710 2024-03-25T17:49:36.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.273+0000 7f7ed27fc640 10 client.4689.objecter ms_dispatch 0x7f7edc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:49:36.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.273+0000 7f7ed27fc640 3 client.4689.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:49:36.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.273+0000 7f7ed27fc640 3 client.4689.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:49:36.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.273+0000 7f7ed27fc640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T17:49:36.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.273+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ecc050400 con 0x7f7edc13f710 2024-03-25T17:49:36.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.273+0000 7f7ed27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:36.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.273+0000 7f7ed27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:49:36.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.505+0000 7f7ee421e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:36.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.505+0000 7f7ee421e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:36.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.505+0000 7f7ee421e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T17:49:36.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.505+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7edc0b7180 con 0x7f7edc13f710 2024-03-25T17:49:36.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.526+0000 7f7ed27fc640 1 -- 172.21.15.44:0/901287447 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7f7ecc052330 con 0x7f7edc13f710 2024-03-25T17:49:36.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.526+0000 7f7ed27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:36.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.526+0000 7f7ed27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:49:36.527 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:49:36.527 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.528+0000 7f7ee421e640 10 librados: watch_flush enter 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.528+0000 7f7ee421e640 10 librados: watch_flush exit 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.528+0000 7f7ee421e640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.528+0000 7f7ee421e640 10 client.4689.objecter successfully canceled tick 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.528+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 >> v1:172.21.15.44:6800/3257043132 conn(0x7f7ec006df70 legacy=0x7f7ec0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.529+0000 7f7ee421e640 10 monclient: shutdown 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.529+0000 7f7ee421e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.529+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 >> v1:172.21.15.44:6789/0 conn(0x7f7edc13f710 legacy=0x7f7edc070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.529+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 shutdown_connections 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.529+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 >> 172.21.15.44:0/901287447 conn(0x7f7edc0b7ea0 msgr2=0x7f7edc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.529+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 shutdown_connections 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.529+0000 7f7ee421e640 1 -- 172.21.15.44:0/901287447 wait complete. 2024-03-25T17:49:36.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:36.529+0000 7f7ee421e640 1 librados: shutdown 2024-03-25T17:49:36.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:49:39.334 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:56.689 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:39.393 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:56.749 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:39.427 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:56.782 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:39.437 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:56.793 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:39.449 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:56.805 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:39.494 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:56.850 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:39.568 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:56.887 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:39.572 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:56.892 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:39.607 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:56.927 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:43.346 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.464+0000 7f7a127af640 10 monclient: get_monmap_and_config 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.464+0000 7f7a127af640 10 monclient: build_initial_monmap 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.464+0000 7f7a127af640 10 monclient: monmap: 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:49:43.465462+0000 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:49:43.465462+0000 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:43.465 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:49:43.466 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:49:43.466 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.465+0000 7f7a127af640 1 Processor -- start 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.465+0000 7f7a127af640 1 -- start start 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.465+0000 7f7a127af640 10 monclient: init 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.465+0000 7f7a127af640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.465+0000 7f7a127af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.465+0000 7f7a127af640 10 monclient(hunting): picked mon.noname-a con 0x7f7a0c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.466+0000 7f7a127af640 10 monclient(hunting): picked mon.noname-c con 0x7f7a0c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.466+0000 7f7a127af640 10 monclient(hunting): picked mon.noname-b con 0x7f7a0c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.466+0000 7f7a127af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c142520 con 0x7f7a0c0a98f0 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.466+0000 7f7a127af640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c0b28a0 con 0x7f7a0c13f710 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.466+0000 7f7a127af640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c0b2a30 con 0x7f7a0c1430c0 2024-03-25T17:49:43.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.466+0000 7f7a127af640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:43.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.466+0000 7f7a127af640 10 monclient(hunting): authenticate will time out at 6586.693767s 2024-03-25T17:49:43.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.481+0000 7f7a0b7fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7a0c13f710 0x7f7a0c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:42764/0 (socket says 172.21.15.44:42764) 2024-03-25T17:49:43.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.481+0000 7f7a0b7fe640 1 -- 172.21.15.44:0/1438241031 learned_addr learned my addr 172.21.15.44:0/1438241031 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:43.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.489+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375373535 0 0) 0x7f7a0c0b28a0 con 0x7f7a0c13f710 2024-03-25T17:49:43.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.489+0000 7f7a0affd640 10 monclient(hunting): _negotiate 2024-03-25T17:49:43.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.489+0000 7f7a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:43.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.489+0000 7f7a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:43.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.489+0000 7f7a0affd640 10 monclient(hunting): my global_id is 4687 2024-03-25T17:49:43.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.489+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f4003520 con 0x7f7a0c13f710 2024-03-25T17:49:43.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.503+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957747201 0 0) 0x7f7a0c0b2a30 con 0x7f7a0c1430c0 2024-03-25T17:49:43.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.503+0000 7f7a0affd640 10 monclient(hunting): _negotiate 2024-03-25T17:49:43.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.503+0000 7f7a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:43.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.503+0000 7f7a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:43.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.503+0000 7f7a0affd640 10 monclient(hunting): my global_id is 4688 2024-03-25T17:49:43.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.503+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a0c0b28a0 con 0x7f7a0c1430c0 2024-03-25T17:49:43.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.505+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3791121358 0 0) 0x7f79f4003520 con 0x7f7a0c13f710 2024-03-25T17:49:43.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 >> v1:172.21.15.44:6790/0 conn(0x7f7a0c1430c0 legacy=0x7f7a0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:43.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 >> v1:172.21.15.44:6789/0 conn(0x7f7a0c0a98f0 legacy=0x7f7a0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 10 monclient: _finish_hunting 0 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 1 monclient: found mon.noname-c 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c0b44c0 con 0x7f7a0c13f710 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 10 monclient: _finish_auth 0 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 10 monclient: _check_auth_tickets 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a00001690 con 0x7f7a0c13f710 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a0affd640 10 monclient: dump: 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:43.507 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a127af640 5 monclient: authenticate success, global_id 4687 2024-03-25T17:49:43.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.506+0000 7f7a127af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:43.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.542+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a000017f0 con 0x7f7a0c13f710 2024-03-25T17:49:43.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.542+0000 7f7a0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.542+0000 7f7a0affd640 1 -- 172.21.15.44:0/1438241031 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a00001c00 con 0x7f7a0c13f710 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.542+0000 7f7a127af640 10 monclient: get_monmap_and_config success 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.542+0000 7f7a127af640 4 set_mon_vals no callback set 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.542+0000 7f7a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.542+0000 7f7a0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.542+0000 7f7a0affd640 10 monclient: dump: 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:43.543 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:43.544 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 10 monclient: shutdown 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 1 -- 172.21.15.44:0/1438241031 >> v1:172.21.15.111:6789/0 conn(0x7f7a0c13f710 legacy=0x7f7a0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 1 -- 172.21.15.44:0/1438241031 shutdown_connections 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 1 -- 172.21.15.44:0/1438241031 >> 172.21.15.44:0/1438241031 conn(0x7f7a0c0b7ea0 msgr2=0x7f7a0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 1 -- 172.21.15.44:0/1438241031 shutdown_connections 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 1 -- 172.21.15.44:0/1438241031 wait complete. 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 10 monclient: build_initial_monmap 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.543+0000 7f7a127af640 10 monclient: monmap: 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:43.545 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.544+0000 7f7a127af640 1 librados: starting msgr at 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.544+0000 7f7a127af640 1 librados: starting objecter 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.544+0000 7f7a127af640 1 Processor -- start 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.544+0000 7f7a127af640 1 -- start start 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.544+0000 7f7a127af640 1 librados: setting wanted keys 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.544+0000 7f7a127af640 1 librados: calling monclient init 2024-03-25T17:49:43.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.544+0000 7f7a127af640 10 monclient: init 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 10 monclient(hunting): picked mon.noname-a con 0x7f7a0c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 10 monclient(hunting): picked mon.noname-c con 0x7f7a0c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 10 monclient(hunting): picked mon.noname-b con 0x7f7a0c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c139070 con 0x7f7a0c0a98f0 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c133710 con 0x7f7a0c1430c0 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c1338a0 con 0x7f7a0c13f710 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:43.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.545+0000 7f7a127af640 10 monclient(hunting): authenticate will time out at 6586.773290s 2024-03-25T17:49:43.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.552+0000 7f7a10d4c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7a0c1430c0 0x7f7a0c133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:42802/0 (socket says 172.21.15.44:42802) 2024-03-25T17:49:43.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.552+0000 7f7a10d4c640 1 -- 172.21.15.44:0/2290975393 learned_addr learned my addr 172.21.15.44:0/2290975393 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:43.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.555+0000 7f7a08ff9640 10 client.?.objecter ms_handle_connect 0x7f7a0c1430c0 2024-03-25T17:49:43.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.555+0000 7f7a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:43.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.560+0000 7f7a08ff9640 10 client.?.objecter ms_handle_connect 0x7f7a0c0a98f0 2024-03-25T17:49:43.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.560+0000 7f7a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:43.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.564+0000 7f7a08ff9640 10 client.?.objecter ms_handle_connect 0x7f7a0c13f710 2024-03-25T17:49:43.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.564+0000 7f7a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:43.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.564+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857144960 0 0) 0x7f7a0c133710 con 0x7f7a0c1430c0 2024-03-25T17:49:43.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.564+0000 7f7a08ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:49:43.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.564+0000 7f7a08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:43.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.564+0000 7f7a08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:43.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.564+0000 7f7a08ff9640 10 monclient(hunting): my global_id is 4690 2024-03-25T17:49:43.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.565+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f8004eb0 con 0x7f7a0c1430c0 2024-03-25T17:49:43.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.574+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138554808 0 0) 0x7f7a0c139070 con 0x7f7a0c0a98f0 2024-03-25T17:49:43.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.574+0000 7f7a08ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:49:43.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.574+0000 7f7a08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:43.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.574+0000 7f7a08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:43.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.574+0000 7f7a08ff9640 10 monclient(hunting): my global_id is 4695 2024-03-25T17:49:43.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.574+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a0c133710 con 0x7f7a0c0a98f0 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.585+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4165231613 0 0) 0x7f79f8004eb0 con 0x7f7a0c1430c0 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.585+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a0c139070 con 0x7f7a0c1430c0 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.585+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a04001c70 con 0x7f7a0c1430c0 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.585+0000 7f7a08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.585+0000 7f7a08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.585+0000 7f7a08ff9640 10 monclient(hunting): dump: 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.586 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.587 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:43.587 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:43.587 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:43.587 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:43.587 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:43.587 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:43.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.585+0000 7f7a08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:49:43.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.595+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740280552 0 0) 0x7f7a0c1338a0 con 0x7f7a0c13f710 2024-03-25T17:49:43.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.595+0000 7f7a08ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:49:43.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.595+0000 7f7a08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:43.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.595+0000 7f7a08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:43.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.595+0000 7f7a08ff9640 10 monclient(hunting): my global_id is 4691 2024-03-25T17:49:43.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.595+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f8004eb0 con 0x7f7a0c13f710 2024-03-25T17:49:43.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.602+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3842932627 0 0) 0x7f7a0c133710 con 0x7f7a0c0a98f0 2024-03-25T17:49:43.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.602+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a0c1338a0 con 0x7f7a0c0a98f0 2024-03-25T17:49:43.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.602+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f79fc0018b0 con 0x7f7a0c0a98f0 2024-03-25T17:49:43.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.602+0000 7f7a08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:43.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.602+0000 7f7a08ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:49:43.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.602+0000 7f7a08ff9640 10 monclient(hunting): dump: 2024-03-25T17:49:43.602 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:43.603 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2643206049 0 0) 0x7f7a0c139070 con 0x7f7a0c1430c0 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 >> v1:172.21.15.44:6790/0 conn(0x7f7a0c13f710 legacy=0x7f7a0c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 >> v1:172.21.15.44:6789/0 conn(0x7f7a0c0a98f0 legacy=0x7f7a0c1378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 1 monclient: found mon.b 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c133a30 con 0x7f7a0c1430c0 2024-03-25T17:49:43.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 10 monclient: _finish_auth 0 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 5 monclient: authenticate success, global_id 4690 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 10 monclient: _renew_subs 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0c0b7100 con 0x7f7a0c1430c0 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 10 monclient: _renew_subs 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0c136e60 con 0x7f7a0c1430c0 2024-03-25T17:49:43.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.607+0000 7f7a127af640 1 librados: init done 2024-03-25T17:49:43.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.608+0000 7f7a127af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:43.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.608+0000 7f7a127af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:43.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.608+0000 7f7a127af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:43.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.608+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0c0627b0 con 0x7f7a0c1430c0 2024-03-25T17:49:43.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.906+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a04003990 con 0x7f7a0c1430c0 2024-03-25T17:49:43.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.906+0000 7f7a08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:43.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.906+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7a04003ce0 con 0x7f7a0c1430c0 2024-03-25T17:49:43.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.906+0000 7f7a08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.907+0000 7f7a08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.907+0000 7f7a08ff9640 10 monclient: dump: 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.906+0000 7f7a0a7fc640 4 set_mon_vals no callback set 2024-03-25T17:49:43.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.907+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f7a04003ec0 con 0x7f7a0c1430c0 2024-03-25T17:49:43.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.908+0000 7f7a0bfff640 10 monclient: get_auth_request con 0x7f79f806dfb0 auth_method 0 2024-03-25T17:49:43.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.908+0000 7f7a08ff9640 10 client.4690.objecter ms_handle_connect 0x7f79f806dfb0 2024-03-25T17:49:43.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.913+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7f7a04054b00 con 0x7f7a0c1430c0 2024-03-25T17:49:43.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.913+0000 7f7a08ff9640 10 client.4690.objecter ms_dispatch 0x7f7a0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:49:43.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.913+0000 7f7a08ff9640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:49:43.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.913+0000 7f7a08ff9640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:49:43.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.913+0000 7f7a08ff9640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T17:49:43.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.914+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a04050610 con 0x7f7a0c1430c0 2024-03-25T17:49:43.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.914+0000 7f7a08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:43.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:43.914+0000 7f7a08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:49:44.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.170+0000 7f7a127af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:44.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.171+0000 7f7a127af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:44.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.171+0000 7f7a127af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:49:44.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.171+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a0c074cc0 con 0x7f7a0c1430c0 2024-03-25T17:49:44.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.321+0000 7f7a08ff9640 1 -- 172.21.15.44:0/2290975393 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7f7a04052540 con 0x7f7a0c1430c0 2024-03-25T17:49:44.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.322+0000 7f7a08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:44.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.322+0000 7f7a08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:49:44.323 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:49:44.323 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:49:44.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 10 librados: watch_flush enter 2024-03-25T17:49:44.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 10 librados: watch_flush exit 2024-03-25T17:49:44.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T17:49:44.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 10 client.4690.objecter successfully canceled tick 2024-03-25T17:49:44.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 >> v1:172.21.15.44:6800/3257043132 conn(0x7f79f806dfb0 legacy=0x7f79f8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:44.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 10 monclient: shutdown 2024-03-25T17:49:44.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:44.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 >> v1:172.21.15.111:6789/0 conn(0x7f7a0c1430c0 legacy=0x7f7a0c133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:44.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 shutdown_connections 2024-03-25T17:49:44.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 >> 172.21.15.44:0/2290975393 conn(0x7f7a0c0b7ea0 msgr2=0x7f7a0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:44.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 shutdown_connections 2024-03-25T17:49:44.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 1 -- 172.21.15.44:0/2290975393 wait complete. 2024-03-25T17:49:44.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:44.324+0000 7f7a127af640 1 librados: shutdown 2024-03-25T17:49:44.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:49:51.141 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:49:51.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.244+0000 7f04d3739640 10 monclient: get_monmap_and_config 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.244+0000 7f04d3739640 10 monclient: build_initial_monmap 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.244+0000 7f04d3739640 10 monclient: monmap: 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:49:51.246148+0000 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:49:51.246148+0000 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:49:51.246 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.244+0000 7f04d3739640 1 Processor -- start 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 1 -- start start 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 10 monclient: init 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 10 monclient(hunting): picked mon.noname-a con 0x7f04cc0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 10 monclient(hunting): picked mon.noname-b con 0x7f04cc0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 10 monclient(hunting): picked mon.noname-c con 0x7f04cc14be90 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0af940 con 0x7f04cc0b4d60 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0b0a70 con 0x7f04cc14be90 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0b0d50 con 0x7f04cc0b1730 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:51.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.245+0000 7f04d3739640 10 monclient(hunting): authenticate will time out at 6594.474450s 2024-03-25T17:49:51.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.252+0000 7f04d1cd6640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f04cc14be90 0x7f04cc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:38936/0 (socket says 172.21.15.44:38936) 2024-03-25T17:49:51.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.252+0000 7f04d1cd6640 1 -- 172.21.15.44:0/3676761422 learned_addr learned my addr 172.21.15.44:0/3676761422 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:51.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.271+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033830529 0 0) 0x7f04cc0b0a70 con 0x7f04cc14be90 2024-03-25T17:49:51.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.271+0000 7f04c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:49:51.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.271+0000 7f04c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:51.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.271+0000 7f04c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:51.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.271+0000 7f04c3fff640 10 monclient(hunting): my global_id is 4693 2024-03-25T17:49:51.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.271+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b8003520 con 0x7f04cc14be90 2024-03-25T17:49:51.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.291+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080388197 0 0) 0x7f04cc0af940 con 0x7f04cc0b4d60 2024-03-25T17:49:51.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.291+0000 7f04c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:49:51.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.291+0000 7f04c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:51.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.291+0000 7f04c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:51.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.291+0000 7f04c3fff640 10 monclient(hunting): my global_id is 4698 2024-03-25T17:49:51.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.291+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc0b0a70 con 0x7f04cc0b4d60 2024-03-25T17:49:51.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1431435462 0 0) 0x7f04b8003520 con 0x7f04cc14be90 2024-03-25T17:49:51.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 >> v1:172.21.15.44:6790/0 conn(0x7f04cc0b1730 legacy=0x7f04cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 >> v1:172.21.15.44:6789/0 conn(0x7f04cc0b4d60 legacy=0x7f04cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 1 monclient: found mon.noname-c 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0b1030 con 0x7f04cc14be90 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 10 monclient: _finish_auth 0 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f04c4001a10 con 0x7f04cc14be90 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.292+0000 7f04c3fff640 10 monclient: dump: 2024-03-25T17:49:51.295 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.293+0000 7f04d3739640 5 monclient: authenticate success, global_id 4693 2024-03-25T17:49:51.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.293+0000 7f04d3739640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.306+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04c4001b70 con 0x7f04cc14be90 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.306+0000 7f04c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.306+0000 7f04c3fff640 1 -- 172.21.15.44:0/3676761422 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f04c4001f80 con 0x7f04cc14be90 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.306+0000 7f04c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.306+0000 7f04c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.306+0000 7f04c3fff640 10 monclient: dump: 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:51.308 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:51.309 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:51.309 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:51.309 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:51.309 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:51.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.306+0000 7f04d3739640 10 monclient: get_monmap_and_config success 2024-03-25T17:49:51.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.306+0000 7f04d3739640 4 set_mon_vals no callback set 2024-03-25T17:49:51.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.307+0000 7f04d3739640 10 monclient: shutdown 2024-03-25T17:49:51.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.307+0000 7f04d3739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:51.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.307+0000 7f04d3739640 1 -- 172.21.15.44:0/3676761422 >> v1:172.21.15.111:6789/0 conn(0x7f04cc14be90 legacy=0x7f04cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:51.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.307+0000 7f04d3739640 1 -- 172.21.15.44:0/3676761422 shutdown_connections 2024-03-25T17:49:51.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.307+0000 7f04d3739640 1 -- 172.21.15.44:0/3676761422 >> 172.21.15.44:0/3676761422 conn(0x7f04cc0617a0 msgr2=0x7f04cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.307+0000 7f04d3739640 1 -- 172.21.15.44:0/3676761422 shutdown_connections 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.307+0000 7f04d3739640 1 -- 172.21.15.44:0/3676761422 wait complete. 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 10 monclient: build_initial_monmap 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 10 monclient: monmap: 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:51.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 1 librados: starting msgr at 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 1 librados: starting objecter 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 1 Processor -- start 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 1 -- start start 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 1 librados: setting wanted keys 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 1 librados: calling monclient init 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.308+0000 7f04d3739640 10 monclient: init 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 10 monclient(hunting): picked mon.noname-a con 0x7f04cc0b1730 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 10 monclient(hunting): picked mon.noname-b con 0x7f04cc0b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 10 monclient(hunting): picked mon.noname-c con 0x7f04cc14be90 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc1363f0 con 0x7f04cc0b1730 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc1374e0 con 0x7f04cc0b4d60 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc138140 con 0x7f04cc14be90 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:51.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.309+0000 7f04d3739640 10 monclient(hunting): authenticate will time out at 6594.538284s 2024-03-25T17:49:51.318 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.317+0000 7f04d1cd6640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f04cc14be90 0x7f04cc143c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:42806/0 (socket says 172.21.15.44:42806) 2024-03-25T17:49:51.318 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.317+0000 7f04d1cd6640 1 -- 172.21.15.44:0/1225136628 learned_addr learned my addr 172.21.15.44:0/1225136628 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:51.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.319+0000 7f04c1ffb640 10 client.?.objecter ms_handle_connect 0x7f04cc14be90 2024-03-25T17:49:51.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.319+0000 7f04c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:51.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.326+0000 7f04c1ffb640 10 client.?.objecter ms_handle_connect 0x7f04cc0b4d60 2024-03-25T17:49:51.327 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.326+0000 7f04c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:51.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.327+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054401145 0 0) 0x7f04cc138140 con 0x7f04cc14be90 2024-03-25T17:49:51.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.327+0000 7f04c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:49:51.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.327+0000 7f04c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:51.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.327+0000 7f04c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:51.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.327+0000 7f04c1ffb640 10 monclient(hunting): my global_id is 4697 2024-03-25T17:49:51.328 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.327+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b0004eb0 con 0x7f04cc14be90 2024-03-25T17:49:51.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.336+0000 7f04c1ffb640 10 client.?.objecter ms_handle_connect 0x7f04cc0b1730 2024-03-25T17:49:51.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.336+0000 7f04c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:51.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.336+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560241115 0 0) 0x7f04cc1374e0 con 0x7f04cc0b4d60 2024-03-25T17:49:51.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.336+0000 7f04c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:49:51.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.336+0000 7f04c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:51.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.336+0000 7f04c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:51.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.336+0000 7f04c1ffb640 10 monclient(hunting): my global_id is 4696 2024-03-25T17:49:51.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.336+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc138140 con 0x7f04cc0b4d60 2024-03-25T17:49:51.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716507681 0 0) 0x7f04cc1363f0 con 0x7f04cc0b1730 2024-03-25T17:49:51.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:49:51.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:51.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:51.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 10 monclient(hunting): my global_id is 4701 2024-03-25T17:49:51.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc1374e0 con 0x7f04cc0b1730 2024-03-25T17:49:51.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117588974 0 0) 0x7f04b0004eb0 con 0x7f04cc14be90 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04cc1363f0 con 0x7f04cc14be90 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f04c4001ec0 con 0x7f04cc14be90 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 10 monclient(hunting): dump: 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:51.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.349+0000 7f04c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:49:51.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.355+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 116945528 0 0) 0x7f04cc138140 con 0x7f04cc0b4d60 2024-03-25T17:49:51.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.355+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04b0004eb0 con 0x7f04cc0b4d60 2024-03-25T17:49:51.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.355+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f04bc0018a0 con 0x7f04cc0b4d60 2024-03-25T17:49:51.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.355+0000 7f04c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:51.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.355+0000 7f04c1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:51.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.355+0000 7f04c1ffb640 10 monclient(hunting): dump: 2024-03-25T17:49:51.357 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:51.357 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:51.358 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.358 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.358 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:51.358 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:51.358 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:51.358 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:51.358 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:51.358 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:51.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.362+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1724751596 0 0) 0x7f04cc1374e0 con 0x7f04cc0b1730 2024-03-25T17:49:51.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.362+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04cc138140 con 0x7f04cc0b1730 2024-03-25T17:49:51.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.362+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f04c80019b0 con 0x7f04cc0b1730 2024-03-25T17:49:51.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.362+0000 7f04c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:51.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.362+0000 7f04c1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.362+0000 7f04c1ffb640 10 monclient(hunting): dump: 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:51.365 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:51.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3681541350 0 0) 0x7f04cc1363f0 con 0x7f04cc14be90 2024-03-25T17:49:51.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 >> v1:172.21.15.111:6789/0 conn(0x7f04cc0b4d60 legacy=0x7f04cc12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:51.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 >> v1:172.21.15.44:6789/0 conn(0x7f04cc0b1730 legacy=0x7f04cc136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:51.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:49:51.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 1 monclient: found mon.c 2024-03-25T17:49:51.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:51.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc139090 con 0x7f04cc14be90 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 5 monclient: authenticate success, global_id 4697 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 10 monclient: _renew_subs 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04cc136030 con 0x7f04cc14be90 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 10 monclient: _renew_subs 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:51.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc130650 con 0x7f04cc14be90 2024-03-25T17:49:51.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.372+0000 7f04d3739640 1 librados: init done 2024-03-25T17:49:51.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.373+0000 7f04d3739640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:51.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.373+0000 7f04d3739640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:51.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.373+0000 7f04d3739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:51.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.373+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04cc0ae6c0 con 0x7f04cc14be90 2024-03-25T17:49:51.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.684+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04c4004e10 con 0x7f04cc14be90 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.684+0000 7f04c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.684+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f04c4002100 con 0x7f04cc14be90 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.684+0000 7f04c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.684+0000 7f04c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.684+0000 7f04c1ffb640 10 monclient: dump: 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:51.686 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:51.687 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:51.687 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:51.687 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:51.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.684+0000 7f04c37fe640 4 set_mon_vals no callback set 2024-03-25T17:49:51.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.685+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f04c401af90 con 0x7f04cc14be90 2024-03-25T17:49:51.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.685+0000 7f04d14d5640 10 monclient: get_auth_request con 0x7f04b006e0f0 auth_method 0 2024-03-25T17:49:51.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.686+0000 7f04c1ffb640 10 client.4697.objecter ms_handle_connect 0x7f04b006e0f0 2024-03-25T17:49:51.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.690+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7f04c4054ab0 con 0x7f04cc14be90 2024-03-25T17:49:51.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.690+0000 7f04c1ffb640 10 client.4697.objecter ms_dispatch 0x7f04cc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:49:51.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.690+0000 7f04c1ffb640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:49:51.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.690+0000 7f04c1ffb640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:49:51.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.690+0000 7f04c1ffb640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-25T17:49:51.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.690+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04c40503f0 con 0x7f04cc14be90 2024-03-25T17:49:51.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.690+0000 7f04c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:51.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.690+0000 7f04c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:49:51.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.946+0000 7f04d3739640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:51.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.946+0000 7f04d3739640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:51.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.946+0000 7f04d3739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:51.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.946+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04cc0627b0 con 0x7f04cc14be90 2024-03-25T17:49:51.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.997+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1225136628 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7f04c40524f0 con 0x7f04cc14be90 2024-03-25T17:49:51.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.997+0000 7f04c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:51.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:51.997+0000 7f04c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:49:51.999 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:49:51.999 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 10 librados: watch_flush enter 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 10 librados: watch_flush exit 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 10 client.4697.objecter successfully canceled tick 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 >> v1:172.21.15.44:6800/3257043132 conn(0x7f04b006e0f0 legacy=0x7f04b00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 10 monclient: shutdown 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 >> v1:172.21.15.44:6790/0 conn(0x7f04cc14be90 legacy=0x7f04cc143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:52.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 shutdown_connections 2024-03-25T17:49:52.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 >> 172.21.15.44:0/1225136628 conn(0x7f04cc0617a0 msgr2=0x7f04cc13c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:52.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 shutdown_connections 2024-03-25T17:49:52.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 1 -- 172.21.15.44:0/1225136628 wait complete. 2024-03-25T17:49:52.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:52.000+0000 7f04d3739640 1 librados: shutdown 2024-03-25T17:49:52.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:49:58.820 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:49:58.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.934+0000 7fa50714d640 10 monclient: get_monmap_and_config 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.934+0000 7fa50714d640 10 monclient: build_initial_monmap 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.934+0000 7fa50714d640 10 monclient: monmap: 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:49:58.935149+0000 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:49:58.935149+0000 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:49:58.935 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.935+0000 7fa50714d640 1 Processor -- start 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.935+0000 7fa50714d640 1 -- start start 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.935+0000 7fa50714d640 10 monclient: init 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5000a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 10 monclient(hunting): picked mon.noname-a con 0x7fa50013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5001430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa500142520 con 0x7fa50013f710 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5000b28a0 con 0x7fa5000a98f0 2024-03-25T17:49:58.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5000b2a30 con 0x7fa5001430c0 2024-03-25T17:49:58.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:58.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.936+0000 7fa50714d640 10 monclient(hunting): authenticate will time out at 6602.163835s 2024-03-25T17:49:58.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.944+0000 7fa504ee9640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa5000a98f0 0x7fa5000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:43778/0 (socket says 172.21.15.44:43778) 2024-03-25T17:49:58.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.944+0000 7fa504ee9640 1 -- 172.21.15.44:0/2314485041 learned_addr learned my addr 172.21.15.44:0/2314485041 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:58.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.958+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215895792 0 0) 0x7fa5000b28a0 con 0x7fa5000a98f0 2024-03-25T17:49:58.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.958+0000 7fa4ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:49:58.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.958+0000 7fa4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:58.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.958+0000 7fa4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:58.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.958+0000 7fa4ff7fe640 10 monclient(hunting): my global_id is 4699 2024-03-25T17:49:58.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.958+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f0003520 con 0x7fa5000a98f0 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973336749 0 0) 0x7fa500142520 con 0x7fa50013f710 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 10 monclient(hunting): my global_id is 4704 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f0003840 con 0x7fa50013f710 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335058745 0 0) 0x7fa5000b2a30 con 0x7fa5001430c0 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:58.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 10 monclient(hunting): my global_id is 4700 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.965+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa500142520 con 0x7fa5001430c0 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3340789321 0 0) 0x7fa4f0003520 con 0x7fa5000a98f0 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 >> v1:172.21.15.44:6790/0 conn(0x7fa5001430c0 legacy=0x7fa50014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 >> v1:172.21.15.44:6789/0 conn(0x7fa50013f710 legacy=0x7fa5001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 1 monclient: found mon.noname-c 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5000b44c0 con 0x7fa5000a98f0 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 10 monclient: _finish_auth 0 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:49:58.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa4f4001670 con 0x7fa5000a98f0 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa4ff7fe640 10 monclient: dump: 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:58.969 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:58.970 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:58.970 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:58.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa50714d640 5 monclient: authenticate success, global_id 4699 2024-03-25T17:49:58.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.966+0000 7fa50714d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:49:58.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.975+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4f40017d0 con 0x7fa5000a98f0 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.975+0000 7fa4ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.975+0000 7fa4ff7fe640 1 -- 172.21.15.44:0/2314485041 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa4f4001be0 con 0x7fa5000a98f0 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.975+0000 7fa4ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.975+0000 7fa4ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.975+0000 7fa4ff7fe640 10 monclient: dump: 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:58.976 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:58.977 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:58.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.975+0000 7fa50714d640 10 monclient: get_monmap_and_config success 2024-03-25T17:49:58.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.975+0000 7fa50714d640 4 set_mon_vals no callback set 2024-03-25T17:49:58.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.976+0000 7fa50714d640 10 monclient: shutdown 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.976+0000 7fa50714d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.976+0000 7fa50714d640 1 -- 172.21.15.44:0/2314485041 >> v1:172.21.15.111:6789/0 conn(0x7fa5000a98f0 legacy=0x7fa5000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.976+0000 7fa50714d640 1 -- 172.21.15.44:0/2314485041 shutdown_connections 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.976+0000 7fa50714d640 1 -- 172.21.15.44:0/2314485041 >> 172.21.15.44:0/2314485041 conn(0x7fa5000b7ea0 msgr2=0x7fa5000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.976+0000 7fa50714d640 1 -- 172.21.15.44:0/2314485041 shutdown_connections 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.976+0000 7fa50714d640 1 -- 172.21.15.44:0/2314485041 wait complete. 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 10 monclient: build_initial_monmap 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 10 monclient: monmap: 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:49:58.979 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 1 librados: starting msgr at 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 1 librados: starting objecter 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 1 Processor -- start 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 1 -- start start 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 1 librados: setting wanted keys 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 1 librados: calling monclient init 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.977+0000 7fa50714d640 10 monclient: init 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 10 monclient: _reopen_session rank -1 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5000a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T17:49:58.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 10 monclient(hunting): picked mon.noname-a con 0x7fa50013f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:49:58.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5001430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:49:58.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5000b7930 con 0x7fa50013f710 2024-03-25T17:49:58.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa500136820 con 0x7fa5000a98f0 2024-03-25T17:49:58.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa500136b00 con 0x7fa5001430c0 2024-03-25T17:49:58.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 10 monclient(hunting): _renew_subs 2024-03-25T17:49:58.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.978+0000 7fa50714d640 10 monclient(hunting): authenticate will time out at 6602.206053s 2024-03-25T17:49:58.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.983+0000 7fa5056ea640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fa5001430c0 0x7fa5001e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:47592/0 (socket says 172.21.15.44:47592) 2024-03-25T17:49:58.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.983+0000 7fa5056ea640 1 -- 172.21.15.44:0/3932369921 learned_addr learned my addr 172.21.15.44:0/3932369921 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:49:58.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.984+0000 7fa4fd7fa640 10 client.?.objecter ms_handle_connect 0x7fa5001430c0 2024-03-25T17:49:58.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.984+0000 7fa4fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:58.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.995+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302006060 0 0) 0x7fa500136b00 con 0x7fa5001430c0 2024-03-25T17:49:58.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.995+0000 7fa4fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:49:58.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.995+0000 7fa4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:58.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.995+0000 7fa4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:58.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.995+0000 7fa4fd7fa640 10 monclient(hunting): my global_id is 4703 2024-03-25T17:49:58.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.995+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e4004eb0 con 0x7fa5001430c0 2024-03-25T17:49:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.998+0000 7fa4fd7fa640 10 client.?.objecter ms_handle_connect 0x7fa5000a98f0 2024-03-25T17:49:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.998+0000 7fa4fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:58.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.999+0000 7fa4fd7fa640 10 client.?.objecter ms_handle_connect 0x7fa50013f710 2024-03-25T17:49:58.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:58.999+0000 7fa4fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:49:59.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.006+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482624688 0 0) 0x7fa5000b7930 con 0x7fa50013f710 2024-03-25T17:49:59.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.006+0000 7fa4fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:49:59.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.006+0000 7fa4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:59.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.006+0000 7fa4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:59.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.006+0000 7fa4fd7fa640 10 monclient(hunting): my global_id is 4707 2024-03-25T17:49:59.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.006+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa500136b00 con 0x7fa50013f710 2024-03-25T17:49:59.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.011+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386322152 0 0) 0x7fa500136820 con 0x7fa5000a98f0 2024-03-25T17:49:59.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.011+0000 7fa4fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:49:59.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.011+0000 7fa4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:49:59.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.011+0000 7fa4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:49:59.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.011+0000 7fa4fd7fa640 10 monclient(hunting): my global_id is 4702 2024-03-25T17:49:59.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.011+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5000b7930 con 0x7fa5000a98f0 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.018+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2798519009 0 0) 0x7fa4e4004eb0 con 0x7fa5001430c0 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.018+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa500136820 con 0x7fa5001430c0 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.018+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa4f8001c90 con 0x7fa5001430c0 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.018+0000 7fa4fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.018+0000 7fa4fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.018+0000 7fa4fd7fa640 10 monclient(hunting): dump: 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:59.019 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:59.020 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:59.020 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:59.020 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:59.020 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:59.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.018+0000 7fa4fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3400473412 0 0) 0x7fa500136820 con 0x7fa5001430c0 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 >> v1:172.21.15.111:6789/0 conn(0x7fa5000a98f0 legacy=0x7fa500135070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 >> v1:172.21.15.44:6789/0 conn(0x7fa50013f710 legacy=0x7fa5001df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 1 monclient: found mon.c 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa500136de0 con 0x7fa5001430c0 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 10 monclient: _finish_auth 0 2024-03-25T17:49:59.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa4fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 5 monclient: authenticate success, global_id 4703 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 10 monclient: _renew_subs 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa500135d80 con 0x7fa5001430c0 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 10 monclient: _renew_subs 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50013b6c0 con 0x7fa5001430c0 2024-03-25T17:49:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.031+0000 7fa50714d640 1 librados: init done 2024-03-25T17:49:59.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.032+0000 7fa50714d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:59.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.032+0000 7fa50714d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:59.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.032+0000 7fa50714d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:59.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.032+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50013d010 con 0x7fa5001430c0 2024-03-25T17:49:59.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.365+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4f8003970 con 0x7fa5001430c0 2024-03-25T17:49:59.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.365+0000 7fa4fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:49:59.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.365+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fa4f8003d40 con 0x7fa5001430c0 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.365+0000 7fa4fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.365+0000 7fa4fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.365+0000 7fa4fd7fa640 10 monclient: dump: 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:49:59.366 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:49:59.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.365+0000 7fa4feffd640 4 set_mon_vals no callback set 2024-03-25T17:49:59.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.365+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fa4f8003f60 con 0x7fa5001430c0 2024-03-25T17:49:59.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.366+0000 7fa504ee9640 10 monclient: get_auth_request con 0x7fa4e406ddf0 auth_method 0 2024-03-25T17:49:59.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.366+0000 7fa4fd7fa640 10 client.4703.objecter ms_handle_connect 0x7fa4e406ddf0 2024-03-25T17:49:59.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.371+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7fa4f8054a90 con 0x7fa5001430c0 2024-03-25T17:49:59.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.371+0000 7fa4fd7fa640 10 client.4703.objecter ms_dispatch 0x7fa500000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:49:59.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.371+0000 7fa4fd7fa640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:49:59.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.371+0000 7fa4fd7fa640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:49:59.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.371+0000 7fa4fd7fa640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T17:49:59.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.371+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4f80505a0 con 0x7fa5001430c0 2024-03-25T17:49:59.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.371+0000 7fa4fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:49:59.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.371+0000 7fa4fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:49:59.445 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:16.801 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.454 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:16.810 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.500 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:16.856 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.506 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:16.861 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.509 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:16.829 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.521 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:16.877 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.530 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:16.850 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.548 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:16.904 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.567 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:16.887 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:49:59.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.595+0000 7fa50714d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:59.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.595+0000 7fa50714d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:59.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.595+0000 7fa50714d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:49:59.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.595+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa500137300 con 0x7fa5001430c0 2024-03-25T17:49:59.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.659+0000 7fa4fd7fa640 1 -- 172.21.15.44:0/3932369921 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7fa500137300 con 0x7fa5001430c0 2024-03-25T17:49:59.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.659+0000 7fa4fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:49:59.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.659+0000 7fa4fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:49:59.660 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:49:59.660 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:49:59.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.661+0000 7fa50714d640 10 librados: watch_flush enter 2024-03-25T17:49:59.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.661+0000 7fa50714d640 10 librados: watch_flush exit 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.661+0000 7fa50714d640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.661+0000 7fa50714d640 10 client.4703.objecter successfully canceled tick 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.661+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 >> v1:172.21.15.44:6800/3257043132 conn(0x7fa4e406ddf0 legacy=0x7fa4e40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.661+0000 7fa50714d640 10 monclient: shutdown 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.661+0000 7fa50714d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.661+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 >> v1:172.21.15.44:6790/0 conn(0x7fa5001430c0 legacy=0x7fa5001e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.662+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 shutdown_connections 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.662+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 >> 172.21.15.44:0/3932369921 conn(0x7fa5000b7ea0 msgr2=0x7fa500074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.662+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 shutdown_connections 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.662+0000 7fa50714d640 1 -- 172.21.15.44:0/3932369921 wait complete. 2024-03-25T17:49:59.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:49:59.662+0000 7fa50714d640 1 librados: shutdown 2024-03-25T17:49:59.675 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:50:02.268 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:19.624 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:02.277 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:19.633 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:02.279 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:19.599 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:02.282 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:19.638 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:02.282 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:19.602 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:02.289 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:19.644 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:02.300 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:19.620 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:02.364 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:19.719 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:02.385 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:19.741 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:06.482 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:50:06.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.596+0000 7fbe9b56b640 10 monclient: get_monmap_and_config 2024-03-25T17:50:06.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.596+0000 7fbe9b56b640 10 monclient: build_initial_monmap 2024-03-25T17:50:06.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.596+0000 7fbe9b56b640 10 monclient: monmap: 2024-03-25T17:50:06.597 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:50:06.597 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:06.597 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:50:06.597130+0000 2024-03-25T17:50:06.597 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:50:06.597130+0000 2024-03-25T17:50:06.597 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:50:06.598 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:06.598 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:50:06.598 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:50:06.598 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:50:06.598 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:06.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.596+0000 7fbe9b56b640 1 Processor -- start 2024-03-25T17:50:06.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 1 -- start start 2024-03-25T17:50:06.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 10 monclient: init 2024-03-25T17:50:06.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 10 monclient: _reopen_session rank -1 2024-03-25T17:50:06.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:50:06.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 10 monclient(hunting): picked mon.noname-c con 0x7fbe9406ea40 addr v1:172.21.15.111:6789/0 2024-03-25T17:50:06.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 10 monclient(hunting): picked mon.noname-b con 0x7fbe94071800 addr v1:172.21.15.44:6790/0 2024-03-25T17:50:06.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 10 monclient(hunting): picked mon.noname-a con 0x7fbe94071c40 addr v1:172.21.15.44:6789/0 2024-03-25T17:50:06.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9406f1f0 con 0x7fbe94071c40 2024-03-25T17:50:06.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe94070d00 con 0x7fbe9406ea40 2024-03-25T17:50:06.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe940720b0 con 0x7fbe94071800 2024-03-25T17:50:06.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 10 monclient(hunting): _renew_subs 2024-03-25T17:50:06.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.597+0000 7fbe9b56b640 10 monclient(hunting): authenticate will time out at 6609.825511s 2024-03-25T17:50:06.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.602+0000 7fbe99b08640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbe94071c40 0x7fbe9414a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42534/0 (socket says 172.21.15.44:42534) 2024-03-25T17:50:06.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.602+0000 7fbe99b08640 1 -- 172.21.15.44:0/1891785387 learned_addr learned my addr 172.21.15.44:0/1891785387 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:50:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.625+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799548669 0 0) 0x7fbe9406f1f0 con 0x7fbe94071c40 2024-03-25T17:50:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.625+0000 7fbe8bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:50:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.625+0000 7fbe8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.625+0000 7fbe8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.625+0000 7fbe8bfff640 10 monclient(hunting): my global_id is 4710 2024-03-25T17:50:06.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.625+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe80003520 con 0x7fbe94071c40 2024-03-25T17:50:06.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.629+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069709349 0 0) 0x7fbe940720b0 con 0x7fbe94071800 2024-03-25T17:50:06.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.629+0000 7fbe8bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:50:06.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.629+0000 7fbe8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:06.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.629+0000 7fbe8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:06.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.629+0000 7fbe8bfff640 10 monclient(hunting): my global_id is 4706 2024-03-25T17:50:06.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.629+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9406f1f0 con 0x7fbe94071800 2024-03-25T17:50:06.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.639+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641075278 0 0) 0x7fbe94070d00 con 0x7fbe9406ea40 2024-03-25T17:50:06.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.639+0000 7fbe8bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:50:06.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.639+0000 7fbe8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:06.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.639+0000 7fbe8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:06.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.639+0000 7fbe8bfff640 10 monclient(hunting): my global_id is 4705 2024-03-25T17:50:06.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.639+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe940720b0 con 0x7fbe9406ea40 2024-03-25T17:50:06.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2923335997 0 0) 0x7fbe80003520 con 0x7fbe94071c40 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 >> v1:172.21.15.44:6790/0 conn(0x7fbe94071800 legacy=0x7fbe94147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 >> v1:172.21.15.111:6789/0 conn(0x7fbe9406ea40 legacy=0x7fbe9413ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 1 monclient: found mon.noname-a 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe94072780 con 0x7fbe94071c40 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 10 monclient: _finish_auth 0 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbe8c001a10 con 0x7fbe94071c40 2024-03-25T17:50:06.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe8bfff640 10 monclient: dump: 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:06.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe9b56b640 5 monclient: authenticate success, global_id 4710 2024-03-25T17:50:06.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.646+0000 7fbe9b56b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:50:06.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.671+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe8c001b70 con 0x7fbe94071c40 2024-03-25T17:50:06.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.671+0000 7fbe8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:50:06.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.671+0000 7fbe8bfff640 1 -- 172.21.15.44:0/1891785387 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbe8c001f80 con 0x7fbe94071c40 2024-03-25T17:50:06.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.671+0000 7fbe9b56b640 10 monclient: get_monmap_and_config success 2024-03-25T17:50:06.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.671+0000 7fbe9b56b640 4 set_mon_vals no callback set 2024-03-25T17:50:06.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.671+0000 7fbe8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:06.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.671+0000 7fbe8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:50:06.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.671+0000 7fbe8bfff640 10 monclient: dump: 2024-03-25T17:50:06.673 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:06.673 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:06.673 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.673 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.673 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:06.673 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:06.673 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:06.673 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:06.674 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:06.674 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:06.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.672+0000 7fbe9b56b640 10 monclient: shutdown 2024-03-25T17:50:06.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.672+0000 7fbe9b56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:50:06.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.672+0000 7fbe9b56b640 1 -- 172.21.15.44:0/1891785387 >> v1:172.21.15.44:6789/0 conn(0x7fbe94071c40 legacy=0x7fbe9414a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:06.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.672+0000 7fbe9b56b640 1 -- 172.21.15.44:0/1891785387 shutdown_connections 2024-03-25T17:50:06.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.672+0000 7fbe9b56b640 1 -- 172.21.15.44:0/1891785387 >> 172.21.15.44:0/1891785387 conn(0x7fbe9406b630 msgr2=0x7fbe940b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:50:06.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.672+0000 7fbe9b56b640 1 -- 172.21.15.44:0/1891785387 shutdown_connections 2024-03-25T17:50:06.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.672+0000 7fbe9b56b640 1 -- 172.21.15.44:0/1891785387 wait complete. 2024-03-25T17:50:06.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 10 monclient: build_initial_monmap 2024-03-25T17:50:06.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 10 monclient: monmap: 2024-03-25T17:50:06.678 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:50:06.678 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:50:06.678 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:50:06.678 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 1 librados: starting msgr at 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 1 librados: starting objecter 2024-03-25T17:50:06.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 1 Processor -- start 2024-03-25T17:50:06.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 1 -- start start 2024-03-25T17:50:06.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 1 librados: setting wanted keys 2024-03-25T17:50:06.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 1 librados: calling monclient init 2024-03-25T17:50:06.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.673+0000 7fbe9b56b640 10 monclient: init 2024-03-25T17:50:06.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 10 monclient: _reopen_session rank -1 2024-03-25T17:50:06.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:50:06.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 10 monclient(hunting): picked mon.noname-c con 0x7fbe9406ea40 addr v1:172.21.15.44:6790/0 2024-03-25T17:50:06.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 10 monclient(hunting): picked mon.noname-a con 0x7fbe94071800 addr v1:172.21.15.44:6789/0 2024-03-25T17:50:06.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 10 monclient(hunting): picked mon.noname-b con 0x7fbe94071c40 addr v1:172.21.15.111:6789/0 2024-03-25T17:50:06.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe94134af0 con 0x7fbe94071800 2024-03-25T17:50:06.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9413b020 con 0x7fbe94071c40 2024-03-25T17:50:06.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9413b300 con 0x7fbe9406ea40 2024-03-25T17:50:06.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 10 monclient(hunting): _renew_subs 2024-03-25T17:50:06.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.674+0000 7fbe9b56b640 10 monclient(hunting): authenticate will time out at 6609.902277s 2024-03-25T17:50:06.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.685+0000 7fbe99b08640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbe94071c40 0x7fbe94136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:48532/0 (socket says 172.21.15.44:48532) 2024-03-25T17:50:06.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.685+0000 7fbe99b08640 1 -- 172.21.15.44:0/766179179 learned_addr learned my addr 172.21.15.44:0/766179179 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:50:06.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.691+0000 7fbe89ffb640 10 client.?.objecter ms_handle_connect 0x7fbe94071c40 2024-03-25T17:50:06.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.691+0000 7fbe89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:06.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.694+0000 7fbe89ffb640 10 client.?.objecter ms_handle_connect 0x7fbe9406ea40 2024-03-25T17:50:06.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.694+0000 7fbe89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:06.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.694+0000 7fbe89ffb640 10 client.?.objecter ms_handle_connect 0x7fbe94071800 2024-03-25T17:50:06.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.694+0000 7fbe89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.698+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346242755 0 0) 0x7fbe94134af0 con 0x7fbe94071800 2024-03-25T17:50:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.698+0000 7fbe89ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:50:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.698+0000 7fbe89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.698+0000 7fbe89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.698+0000 7fbe89ffb640 10 monclient(hunting): my global_id is 4713 2024-03-25T17:50:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.698+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe78004eb0 con 0x7fbe94071800 2024-03-25T17:50:06.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.705+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849597559 0 0) 0x7fbe9413b020 con 0x7fbe94071c40 2024-03-25T17:50:06.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.705+0000 7fbe89ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:50:06.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.705+0000 7fbe89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:06.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.705+0000 7fbe89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:06.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.705+0000 7fbe89ffb640 10 monclient(hunting): my global_id is 4708 2024-03-25T17:50:06.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.705+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe94134af0 con 0x7fbe94071c40 2024-03-25T17:50:06.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.708+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816893190 0 0) 0x7fbe9413b300 con 0x7fbe9406ea40 2024-03-25T17:50:06.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.708+0000 7fbe89ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:50:06.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.708+0000 7fbe89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:06.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.708+0000 7fbe89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:06.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.708+0000 7fbe89ffb640 10 monclient(hunting): my global_id is 4709 2024-03-25T17:50:06.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.708+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9413b020 con 0x7fbe9406ea40 2024-03-25T17:50:06.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.720+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2250191097 0 0) 0x7fbe94134af0 con 0x7fbe94071c40 2024-03-25T17:50:06.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.720+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe9413b300 con 0x7fbe94071c40 2024-03-25T17:50:06.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.720+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbe8c001e80 con 0x7fbe94071c40 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.720+0000 7fbe89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.721+0000 7fbe89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.721+0000 7fbe89ffb640 10 monclient(hunting): dump: 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:06.722 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:06.723 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:06.723 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:06.723 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:06.723 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:06.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.721+0000 7fbe89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:50:06.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.728+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2230156880 0 0) 0x7fbe78004eb0 con 0x7fbe94071800 2024-03-25T17:50:06.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.728+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe94134af0 con 0x7fbe94071800 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.728+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbe84001970 con 0x7fbe94071800 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.728+0000 7fbe89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.728+0000 7fbe89ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.728+0000 7fbe89ffb640 10 monclient(hunting): dump: 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:06.729 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:06.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2382665987 0 0) 0x7fbe9413b300 con 0x7fbe94071c40 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 >> v1:172.21.15.44:6790/0 conn(0x7fbe9406ea40 legacy=0x7fbe941393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 >> v1:172.21.15.44:6789/0 conn(0x7fbe94071800 legacy=0x7fbe94136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 1 monclient: found mon.b 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9413a000 con 0x7fbe94071c40 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 10 monclient: _finish_auth 0 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 5 monclient: authenticate success, global_id 4708 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 10 monclient: _renew_subs 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe94139ab0 con 0x7fbe94071c40 2024-03-25T17:50:06.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:50:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 10 monclient: _renew_subs 2024-03-25T17:50:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe941e0fb0 con 0x7fbe94071c40 2024-03-25T17:50:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.733+0000 7fbe9b56b640 1 librados: init done 2024-03-25T17:50:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.735+0000 7fbe9b56b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.735+0000 7fbe9b56b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:06.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.735+0000 7fbe9b56b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:06.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.735+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9413a240 con 0x7fbe94071c40 2024-03-25T17:50:06.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.752+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe8c001570 con 0x7fbe94071c40 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.752+0000 7fbe89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.752+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fbe8c002080 con 0x7fbe94071c40 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.752+0000 7fbe89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.752+0000 7fbe89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.752+0000 7fbe89ffb640 10 monclient: dump: 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:06.753 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:06.754 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:06.754 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:06.754 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:06.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.752+0000 7fbe8b7fe640 4 set_mon_vals no callback set 2024-03-25T17:50:06.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.753+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fbe8c01acc0 con 0x7fbe94071c40 2024-03-25T17:50:06.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.753+0000 7fbe99307640 10 monclient: get_auth_request con 0x7fbe7806ded0 auth_method 0 2024-03-25T17:50:06.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.754+0000 7fbe89ffb640 10 client.4708.objecter ms_handle_connect 0x7fbe7806ded0 2024-03-25T17:50:06.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.755+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7fbe8c0549e0 con 0x7fbe94071c40 2024-03-25T17:50:06.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.755+0000 7fbe89ffb640 10 client.4708.objecter ms_dispatch 0x7fbe94000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:50:06.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.755+0000 7fbe89ffb640 3 client.4708.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:50:06.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.755+0000 7fbe89ffb640 3 client.4708.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:50:06.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:06.755+0000 7fbe89ffb640 20 client.4708.objecter dump_active .. 0 homeless 2024-03-25T17:50:07.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.010+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe8c050320 con 0x7fbe94071c40 2024-03-25T17:50:07.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.010+0000 7fbe89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:07.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.010+0000 7fbe89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:50:07.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.244+0000 7fbe9b56b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:07.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.244+0000 7fbe9b56b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:07.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.244+0000 7fbe9b56b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:07.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.244+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe940627b0 con 0x7fbe94071c40 2024-03-25T17:50:07.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.275+0000 7fbe89ffb640 1 -- 172.21.15.44:0/766179179 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7fbe8c052420 con 0x7fbe94071c40 2024-03-25T17:50:07.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.275+0000 7fbe89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:07.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.275+0000 7fbe89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:50:07.276 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:50:07.276 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:50:07.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 10 librados: watch_flush enter 2024-03-25T17:50:07.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 10 librados: watch_flush exit 2024-03-25T17:50:07.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 20 client.4708.objecter shutdown clearing up homeless session... 2024-03-25T17:50:07.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 10 client.4708.objecter successfully canceled tick 2024-03-25T17:50:07.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 >> v1:172.21.15.44:6800/3257043132 conn(0x7fbe7806ded0 legacy=0x7fbe78070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:07.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 10 monclient: shutdown 2024-03-25T17:50:07.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:50:07.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 >> v1:172.21.15.111:6789/0 conn(0x7fbe94071c40 legacy=0x7fbe94136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:07.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 shutdown_connections 2024-03-25T17:50:07.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.277+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 >> 172.21.15.44:0/766179179 conn(0x7fbe9406b630 msgr2=0x7fbe94135270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:50:07.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.278+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 shutdown_connections 2024-03-25T17:50:07.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.278+0000 7fbe9b56b640 1 -- 172.21.15.44:0/766179179 wait complete. 2024-03-25T17:50:07.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:07.278+0000 7fbe9b56b640 1 librados: shutdown 2024-03-25T17:50:07.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:50:14.098 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.208+0000 7f8ea96e6640 10 monclient: get_monmap_and_config 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.208+0000 7f8ea96e6640 10 monclient: build_initial_monmap 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.208+0000 7f8ea96e6640 10 monclient: monmap: 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:50:14.209554+0000 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:50:14.209554+0000 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:14.209 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:50:14.210 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:50:14.210 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:50:14.210 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 1 Processor -- start 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 1 -- start start 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 10 monclient: init 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea40a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea413f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea41430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:50:14.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea4142520 con 0x7f8ea40a98f0 2024-03-25T17:50:14.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40b28a0 con 0x7f8ea41430c0 2024-03-25T17:50:14.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40b2a30 con 0x7f8ea413f710 2024-03-25T17:50:14.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:50:14.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.209+0000 7f8ea96e6640 10 monclient(hunting): authenticate will time out at 6617.437652s 2024-03-25T17:50:14.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.210+0000 7f8e9bfff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f8ea413f710 0x7f8ea41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:33730/0 (socket says 172.21.15.44:33730) 2024-03-25T17:50:14.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.210+0000 7f8e9bfff640 1 -- 172.21.15.44:0/892350303 learned_addr learned my addr 172.21.15.44:0/892350303 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:50:14.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.256+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310299766 0 0) 0x7f8ea4142520 con 0x7f8ea40a98f0 2024-03-25T17:50:14.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.256+0000 7f8ea27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:50:14.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.256+0000 7f8ea27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:14.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.256+0000 7f8ea27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:14.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.256+0000 7f8ea27fc640 10 monclient(hunting): my global_id is 4716 2024-03-25T17:50:14.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.256+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e90003520 con 0x7f8ea40a98f0 2024-03-25T17:50:14.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3745274399 0 0) 0x7f8e90003520 con 0x7f8ea40a98f0 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 >> v1:172.21.15.44:6790/0 conn(0x7f8ea413f710 legacy=0x7f8ea41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 >> v1:172.21.15.111:6789/0 conn(0x7f8ea41430c0 legacy=0x7f8ea414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 10 monclient: _finish_hunting 0 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 1 monclient: found mon.noname-a 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea40b44c0 con 0x7f8ea40a98f0 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 10 monclient: _finish_auth 0 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 10 monclient: _check_auth_tickets 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e94001690 con 0x7f8ea40a98f0 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:50:14.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea27fc640 10 monclient: dump: 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea96e6640 5 monclient: authenticate success, global_id 4716 2024-03-25T17:50:14.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.261+0000 7f8ea96e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:50:14.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.290+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e940017f0 con 0x7f8ea40a98f0 2024-03-25T17:50:14.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.290+0000 7f8ea27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.290+0000 7f8ea27fc640 1 -- 172.21.15.44:0/892350303 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e94001c00 con 0x7f8ea40a98f0 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.290+0000 7f8ea96e6640 10 monclient: get_monmap_and_config success 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.290+0000 7f8ea96e6640 4 set_mon_vals no callback set 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.290+0000 7f8ea27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.290+0000 7f8ea27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.290+0000 7f8ea27fc640 10 monclient: dump: 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:14.291 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:14.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.291+0000 7f8ea96e6640 10 monclient: shutdown 2024-03-25T17:50:14.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.291+0000 7f8ea96e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.291+0000 7f8ea96e6640 1 -- 172.21.15.44:0/892350303 >> v1:172.21.15.44:6789/0 conn(0x7f8ea40a98f0 legacy=0x7f8ea40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.291+0000 7f8ea96e6640 1 -- 172.21.15.44:0/892350303 shutdown_connections 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.291+0000 7f8ea96e6640 1 -- 172.21.15.44:0/892350303 >> 172.21.15.44:0/892350303 conn(0x7f8ea40b7ea0 msgr2=0x7f8ea40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.291+0000 7f8ea96e6640 1 -- 172.21.15.44:0/892350303 shutdown_connections 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.291+0000 7f8ea96e6640 1 -- 172.21.15.44:0/892350303 wait complete. 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.292+0000 7f8ea96e6640 10 monclient: build_initial_monmap 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.292+0000 7f8ea96e6640 10 monclient: monmap: 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:50:14.293 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:50:14.294 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:50:14.294 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:14.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.292+0000 7f8ea96e6640 1 librados: starting msgr at 2024-03-25T17:50:14.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.292+0000 7f8ea96e6640 1 librados: starting objecter 2024-03-25T17:50:14.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.292+0000 7f8ea96e6640 1 Processor -- start 2024-03-25T17:50:14.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 1 -- start start 2024-03-25T17:50:14.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 1 librados: setting wanted keys 2024-03-25T17:50:14.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 1 librados: calling monclient init 2024-03-25T17:50:14.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 10 monclient: init 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea40a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea413f710 addr v1:172.21.15.111:6789/0 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea41430c0 addr v1:172.21.15.44:6790/0 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea41d82d0 con 0x7f8ea40a98f0 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.293+0000 7f8ea96e6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40abb90 con 0x7f8ea413f710 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.294+0000 7f8ea96e6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea41d76a0 con 0x7f8ea41430c0 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.294+0000 7f8ea96e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:50:14.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.294+0000 7f8ea96e6640 10 monclient(hunting): authenticate will time out at 6617.521742s 2024-03-25T17:50:14.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.305+0000 7f8ea2ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8ea40a98f0 0x7f8ea41385d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46394/0 (socket says 172.21.15.44:46394) 2024-03-25T17:50:14.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.305+0000 7f8ea2ffd640 1 -- 172.21.15.44:0/3025165867 learned_addr learned my addr 172.21.15.44:0/3025165867 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:50:14.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.309+0000 7f8e9b7fe640 10 client.?.objecter ms_handle_connect 0x7f8ea40a98f0 2024-03-25T17:50:14.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.309+0000 7f8e9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:14.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.312+0000 7f8e9b7fe640 10 client.?.objecter ms_handle_connect 0x7f8ea413f710 2024-03-25T17:50:14.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.312+0000 7f8e9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:14.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.326+0000 7f8e9b7fe640 10 client.?.objecter ms_handle_connect 0x7f8ea41430c0 2024-03-25T17:50:14.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.326+0000 7f8e9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:14.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.329+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555746023 0 0) 0x7f8ea41d82d0 con 0x7f8ea40a98f0 2024-03-25T17:50:14.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.329+0000 7f8e9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:50:14.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.329+0000 7f8e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:14.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.329+0000 7f8e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:14.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.329+0000 7f8e9b7fe640 10 monclient(hunting): my global_id is 4719 2024-03-25T17:50:14.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.329+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e84004eb0 con 0x7f8ea40a98f0 2024-03-25T17:50:14.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.335+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989591353 0 0) 0x7f8ea40abb90 con 0x7f8ea413f710 2024-03-25T17:50:14.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.336+0000 7f8e9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:50:14.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.336+0000 7f8e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:14.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.336+0000 7f8e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:14.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.336+0000 7f8e9b7fe640 10 monclient(hunting): my global_id is 4711 2024-03-25T17:50:14.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.336+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea41d82d0 con 0x7f8ea413f710 2024-03-25T17:50:14.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.339+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890844894 0 0) 0x7f8ea41d76a0 con 0x7f8ea41430c0 2024-03-25T17:50:14.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.339+0000 7f8e9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:50:14.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.339+0000 7f8e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:14.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.339+0000 7f8e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:14.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.339+0000 7f8e9b7fe640 10 monclient(hunting): my global_id is 4715 2024-03-25T17:50:14.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.339+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea40abb90 con 0x7f8ea41430c0 2024-03-25T17:50:14.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.359+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725094924 0 0) 0x7f8e84004eb0 con 0x7f8ea40a98f0 2024-03-25T17:50:14.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.359+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ea41d76a0 con 0x7f8ea40a98f0 2024-03-25T17:50:14.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.359+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e94001690 con 0x7f8ea40a98f0 2024-03-25T17:50:14.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.359+0000 7f8e9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:14.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.359+0000 7f8e9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:50:14.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.359+0000 7f8e9b7fe640 10 monclient(hunting): dump: 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.359+0000 7f8e9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.360+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2492793609 0 0) 0x7f8ea40abb90 con 0x7f8ea41430c0 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.360+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e84004eb0 con 0x7f8ea41430c0 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.360+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e9c001a60 con 0x7f8ea41430c0 2024-03-25T17:50:14.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.360+0000 7f8e9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.360+0000 7f8e9b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.360+0000 7f8e9b7fe640 10 monclient(hunting): dump: 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:14.362 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.361+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375462545 0 0) 0x7f8ea41d82d0 con 0x7f8ea413f710 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.361+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ea40abb90 con 0x7f8ea413f710 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.361+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e8c0018a0 con 0x7f8ea413f710 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.361+0000 7f8e9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.361+0000 7f8e9b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.361+0000 7f8e9b7fe640 10 monclient(hunting): dump: 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.363 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:14.364 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:14.364 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:14.364 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:14.364 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:14.364 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:14.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1495354669 0 0) 0x7f8e84004eb0 con 0x7f8ea41430c0 2024-03-25T17:50:14.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 >> v1:172.21.15.111:6789/0 conn(0x7f8ea413f710 legacy=0x7f8ea41d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 >> v1:172.21.15.44:6789/0 conn(0x7f8ea40a98f0 legacy=0x7f8ea41385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 1 monclient: found mon.c 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4134720 con 0x7f8ea41430c0 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8e9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 5 monclient: authenticate success, global_id 4715 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 10 monclient: _renew_subs 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea4134970 con 0x7f8ea41430c0 2024-03-25T17:50:14.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:50:14.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 10 monclient: _renew_subs 2024-03-25T17:50:14.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:14.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea4137a10 con 0x7f8ea41430c0 2024-03-25T17:50:14.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.374+0000 7f8ea96e6640 1 librados: init done 2024-03-25T17:50:14.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.375+0000 7f8ea96e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:14.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.376+0000 7f8ea96e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:14.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.376+0000 7f8ea96e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:14.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.376+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea4074cc0 con 0x7f8ea41430c0 2024-03-25T17:50:14.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.678+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e9c001f90 con 0x7f8ea41430c0 2024-03-25T17:50:14.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.678+0000 7f8e9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:50:14.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.678+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f8e9c003b10 con 0x7f8ea41430c0 2024-03-25T17:50:14.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.678+0000 7f8e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:14.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.678+0000 7f8e9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:50:14.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.678+0000 7f8e9b7fe640 10 monclient: dump: 2024-03-25T17:50:14.679 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:14.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.678+0000 7f8ea1ffb640 4 set_mon_vals no callback set 2024-03-25T17:50:14.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.679+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f8e9c019ae0 con 0x7f8ea41430c0 2024-03-25T17:50:14.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.680+0000 7f8ea2ffd640 10 monclient: get_auth_request con 0x7f8e8406e110 auth_method 0 2024-03-25T17:50:14.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.680+0000 7f8e9b7fe640 10 client.4715.objecter ms_handle_connect 0x7f8e8406e110 2024-03-25T17:50:14.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.685+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7f8e9c053730 con 0x7f8ea41430c0 2024-03-25T17:50:14.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.685+0000 7f8e9b7fe640 10 client.4715.objecter ms_dispatch 0x7f8ea4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:50:14.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.685+0000 7f8e9b7fe640 3 client.4715.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:50:14.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.685+0000 7f8e9b7fe640 3 client.4715.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:50:14.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.685+0000 7f8e9b7fe640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-25T17:50:14.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.685+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e9c04f070 con 0x7f8ea41430c0 2024-03-25T17:50:14.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.685+0000 7f8e9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:14.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.685+0000 7f8e9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:50:14.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.942+0000 7f8ea96e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:14.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.942+0000 7f8ea96e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:14.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.942+0000 7f8ea96e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:14.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.942+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ea41d7830 con 0x7f8ea41430c0 2024-03-25T17:50:14.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.963+0000 7f8e9b7fe640 1 -- 172.21.15.44:0/3025165867 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7f8e9c051170 con 0x7f8ea41430c0 2024-03-25T17:50:14.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.963+0000 7f8e9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:14.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.963+0000 7f8e9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:50:14.964 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:50:14.964 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:50:14.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 10 librados: watch_flush enter 2024-03-25T17:50:14.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 10 librados: watch_flush exit 2024-03-25T17:50:14.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-25T17:50:14.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 10 client.4715.objecter successfully canceled tick 2024-03-25T17:50:14.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 >> v1:172.21.15.44:6800/3257043132 conn(0x7f8e8406e110 legacy=0x7f8e840705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:14.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 10 monclient: shutdown 2024-03-25T17:50:14.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:50:14.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 >> v1:172.21.15.44:6790/0 conn(0x7f8ea41430c0 legacy=0x7f8ea4134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:14.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 shutdown_connections 2024-03-25T17:50:14.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 >> 172.21.15.44:0/3025165867 conn(0x7f8ea40b7ea0 msgr2=0x7f8ea4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:50:14.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 shutdown_connections 2024-03-25T17:50:14.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 1 -- 172.21.15.44:0/3025165867 wait complete. 2024-03-25T17:50:14.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:14.966+0000 7f8ea96e6640 1 librados: shutdown 2024-03-25T17:50:14.979 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:50:19.516 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:36.872 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:19.570 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:36.926 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:19.574 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:36.930 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:19.603 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:36.958 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:19.632 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:36.952 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:19.635 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:36.991 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:19.651 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:36.971 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:19.660 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:37.016 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:19.695 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:37.015 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:21.789 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:50:21.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.904+0000 7fc257250640 10 monclient: get_monmap_and_config 2024-03-25T17:50:21.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.904+0000 7fc257250640 10 monclient: build_initial_monmap 2024-03-25T17:50:21.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.904+0000 7fc257250640 10 monclient: monmap: 2024-03-25T17:50:21.904 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:50:21.905034+0000 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:50:21.905034+0000 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:50:21.905 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:21.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.904+0000 7fc257250640 1 Processor -- start 2024-03-25T17:50:21.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 1 -- start start 2024-03-25T17:50:21.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 10 monclient: init 2024-03-25T17:50:21.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 10 monclient: _reopen_session rank -1 2024-03-25T17:50:21.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:50:21.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 10 monclient(hunting): picked mon.noname-a con 0x7fc25006c380 addr v1:172.21.15.44:6789/0 2024-03-25T17:50:21.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 10 monclient(hunting): picked mon.noname-b con 0x7fc2500b12d0 addr v1:172.21.15.44:6790/0 2024-03-25T17:50:21.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 10 monclient(hunting): picked mon.noname-c con 0x7fc2500aedf0 addr v1:172.21.15.111:6789/0 2024-03-25T17:50:21.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2500b16f0 con 0x7fc25006c380 2024-03-25T17:50:21.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2500adfe0 con 0x7fc2500aedf0 2024-03-25T17:50:21.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2500add00 con 0x7fc2500b12d0 2024-03-25T17:50:21.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 10 monclient(hunting): _renew_subs 2024-03-25T17:50:21.907 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.905+0000 7fc257250640 10 monclient(hunting): authenticate will time out at 6625.133432s 2024-03-25T17:50:21.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.916+0000 7fc254fec640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc25006c380 0x7fc2500b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41268/0 (socket says 172.21.15.44:41268) 2024-03-25T17:50:21.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.916+0000 7fc254fec640 1 -- 172.21.15.44:0/1531431838 learned_addr learned my addr 172.21.15.44:0/1531431838 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:50:21.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.920+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114553689 0 0) 0x7fc2500b16f0 con 0x7fc25006c380 2024-03-25T17:50:21.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.920+0000 7fc24f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:50:21.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.920+0000 7fc24f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:21.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.920+0000 7fc24f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:21.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.920+0000 7fc24f7fe640 10 monclient(hunting): my global_id is 4722 2024-03-25T17:50:21.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.920+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc240003520 con 0x7fc25006c380 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3152187682 0 0) 0x7fc240003520 con 0x7fc25006c380 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 >> v1:172.21.15.44:6790/0 conn(0x7fc2500b12d0 legacy=0x7fc2500ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 >> v1:172.21.15.111:6789/0 conn(0x7fc2500aedf0 legacy=0x7fc25014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 1 monclient: found mon.noname-a 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2500ae2c0 con 0x7fc25006c380 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc244001690 con 0x7fc25006c380 2024-03-25T17:50:21.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc257250640 5 monclient: authenticate success, global_id 4722 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc257250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc24f7fe640 10 monclient: dump: 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:21.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.937+0000 7fc257250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.955+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2440017f0 con 0x7fc25006c380 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.955+0000 7fc24f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.955+0000 7fc24f7fe640 1 -- 172.21.15.44:0/1531431838 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc244001b80 con 0x7fc25006c380 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.955+0000 7fc257250640 10 monclient: get_monmap_and_config success 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.955+0000 7fc257250640 4 set_mon_vals no callback set 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.955+0000 7fc24f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.955+0000 7fc24f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.955+0000 7fc24f7fe640 10 monclient: dump: 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:21.956 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:21.957 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:21.957 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:21.957 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:21.957 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:21.957 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.956+0000 7fc257250640 10 monclient: shutdown 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.956+0000 7fc257250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.956+0000 7fc257250640 1 -- 172.21.15.44:0/1531431838 >> v1:172.21.15.44:6789/0 conn(0x7fc25006c380 legacy=0x7fc2500b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.956+0000 7fc257250640 1 -- 172.21.15.44:0/1531431838 shutdown_connections 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.956+0000 7fc257250640 1 -- 172.21.15.44:0/1531431838 >> 172.21.15.44:0/1531431838 conn(0x7fc2500b5d10 msgr2=0x7fc2500b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.956+0000 7fc257250640 1 -- 172.21.15.44:0/1531431838 shutdown_connections 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.956+0000 7fc257250640 1 -- 172.21.15.44:0/1531431838 wait complete. 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 10 monclient: build_initial_monmap 2024-03-25T17:50:21.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 10 monclient: monmap: 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 1 librados: starting msgr at 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 1 librados: starting objecter 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 1 Processor -- start 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 1 -- start start 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 1 librados: setting wanted keys 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 1 librados: calling monclient init 2024-03-25T17:50:21.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.957+0000 7fc257250640 10 monclient: init 2024-03-25T17:50:21.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 10 monclient: _reopen_session rank -1 2024-03-25T17:50:21.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:50:21.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 10 monclient(hunting): picked mon.noname-c con 0x7fc25006c380 addr v1:172.21.15.44:6790/0 2024-03-25T17:50:21.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 10 monclient(hunting): picked mon.noname-a con 0x7fc2500aedf0 addr v1:172.21.15.44:6789/0 2024-03-25T17:50:21.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 10 monclient(hunting): picked mon.noname-b con 0x7fc2500b12d0 addr v1:172.21.15.111:6789/0 2024-03-25T17:50:21.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25006c8e0 con 0x7fc2500aedf0 2024-03-25T17:50:21.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2501d8550 con 0x7fc2500b12d0 2024-03-25T17:50:21.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2501c3190 con 0x7fc25006c380 2024-03-25T17:50:21.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 10 monclient(hunting): _renew_subs 2024-03-25T17:50:21.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.958+0000 7fc257250640 10 monclient(hunting): authenticate will time out at 6625.186379s 2024-03-25T17:50:21.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.965+0000 7fc24ffff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fc2500aedf0 0x7fc2501c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41288/0 (socket says 172.21.15.44:41288) 2024-03-25T17:50:21.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.965+0000 7fc24ffff640 1 -- 172.21.15.44:0/3947268221 learned_addr learned my addr 172.21.15.44:0/3947268221 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:50:21.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.971+0000 7fc24d7fa640 10 client.?.objecter ms_handle_connect 0x7fc2500aedf0 2024-03-25T17:50:21.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.971+0000 7fc24d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:21.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.978+0000 7fc24d7fa640 10 client.?.objecter ms_handle_connect 0x7fc25006c380 2024-03-25T17:50:21.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.978+0000 7fc24d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:21.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.979+0000 7fc24d7fa640 10 client.?.objecter ms_handle_connect 0x7fc2500b12d0 2024-03-25T17:50:21.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.979+0000 7fc24d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:21.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.994+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207439372 0 0) 0x7fc2501d8550 con 0x7fc2500b12d0 2024-03-25T17:50:21.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.994+0000 7fc24d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:50:21.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.994+0000 7fc24d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:21.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.994+0000 7fc24d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:21.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.994+0000 7fc24d7fa640 10 monclient(hunting): my global_id is 4717 2024-03-25T17:50:21.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.994+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc234004eb0 con 0x7fc2500b12d0 2024-03-25T17:50:21.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.996+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263882975 0 0) 0x7fc25006c8e0 con 0x7fc2500aedf0 2024-03-25T17:50:21.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.996+0000 7fc24d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:50:21.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.996+0000 7fc24d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:21.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.996+0000 7fc24d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:21.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.996+0000 7fc24d7fa640 10 monclient(hunting): my global_id is 4725 2024-03-25T17:50:21.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:21.996+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2501d8550 con 0x7fc2500aedf0 2024-03-25T17:50:22.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.002+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425013486 0 0) 0x7fc2501c3190 con 0x7fc25006c380 2024-03-25T17:50:22.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.002+0000 7fc24d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:50:22.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.002+0000 7fc24d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:22.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.002+0000 7fc24d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:22.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.002+0000 7fc24d7fa640 10 monclient(hunting): my global_id is 4721 2024-03-25T17:50:22.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.002+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25006c8e0 con 0x7fc25006c380 2024-03-25T17:50:22.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.022+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1460637661 0 0) 0x7fc25006c8e0 con 0x7fc25006c380 2024-03-25T17:50:22.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.022+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2501c3190 con 0x7fc25006c380 2024-03-25T17:50:22.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.022+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc244001960 con 0x7fc25006c380 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.022+0000 7fc24d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.022+0000 7fc24d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.022+0000 7fc24d7fa640 10 monclient(hunting): dump: 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:22.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.022+0000 7fc24d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.025+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2816161004 0 0) 0x7fc234004eb0 con 0x7fc2500b12d0 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.025+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc25006c8e0 con 0x7fc2500b12d0 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.025+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc248001c30 con 0x7fc2500b12d0 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.025+0000 7fc24d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.025+0000 7fc24d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.025+0000 7fc24d7fa640 10 monclient(hunting): dump: 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:22.026 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:22.027 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:22.027 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:22.027 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:22.027 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:22.027 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.030+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 74350232 0 0) 0x7fc2501d8550 con 0x7fc2500aedf0 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.030+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc234004eb0 con 0x7fc2500aedf0 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.030+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc23c001910 con 0x7fc2500aedf0 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.030+0000 7fc24d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.030+0000 7fc24d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.030+0000 7fc24d7fa640 10 monclient(hunting): dump: 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:22.031 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:22.032 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:22.032 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:22.032 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2228296824 0 0) 0x7fc2501c3190 con 0x7fc25006c380 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 >> v1:172.21.15.111:6789/0 conn(0x7fc2500b12d0 legacy=0x7fc2501d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 >> v1:172.21.15.44:6789/0 conn(0x7fc2500aedf0 legacy=0x7fc2501c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 1 monclient: found mon.c 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2501c1980 con 0x7fc25006c380 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 10 monclient: _finish_auth 0 2024-03-25T17:50:22.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc24d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 5 monclient: authenticate success, global_id 4721 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 10 monclient: _renew_subs 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2501c1120 con 0x7fc25006c380 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 10 client.4721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 10 monclient: _renew_subs 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2501c4f40 con 0x7fc25006c380 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.031+0000 7fc257250640 1 librados: init done 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.032+0000 7fc257250640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.032+0000 7fc257250640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.032+0000 7fc257250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:22.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.032+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2501c22e0 con 0x7fc25006c380 2024-03-25T17:50:22.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.344+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc244001b40 con 0x7fc25006c380 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.344+0000 7fc24d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.344+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7fc244004c30 con 0x7fc25006c380 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.344+0000 7fc24d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.344+0000 7fc24d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.344+0000 7fc24d7fa640 10 monclient: dump: 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:22.345 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:22.346 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:22.346 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:22.346 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:22.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.344+0000 7fc24effd640 4 set_mon_vals no callback set 2024-03-25T17:50:22.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.344+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7fc24401ac00 con 0x7fc25006c380 2024-03-25T17:50:22.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.345+0000 7fc24ffff640 10 monclient: get_auth_request con 0x7fc23406e0f0 auth_method 0 2024-03-25T17:50:22.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.345+0000 7fc24d7fa640 10 client.4721.objecter ms_handle_connect 0x7fc23406e0f0 2024-03-25T17:50:22.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.352+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7fc244054700 con 0x7fc25006c380 2024-03-25T17:50:22.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.352+0000 7fc24d7fa640 10 client.4721.objecter ms_dispatch 0x7fc250000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:50:22.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.353+0000 7fc24d7fa640 3 client.4721.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:50:22.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.353+0000 7fc24d7fa640 3 client.4721.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:50:22.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.353+0000 7fc24d7fa640 20 client.4721.objecter dump_active .. 0 homeless 2024-03-25T17:50:22.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.353+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc244050210 con 0x7fc25006c380 2024-03-25T17:50:22.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.353+0000 7fc24d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:22.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.353+0000 7fc24d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:50:22.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.594+0000 7fc257250640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:22.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.594+0000 7fc257250640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:22.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.594+0000 7fc257250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T17:50:22.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.594+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2500627b0 con 0x7fc25006c380 2024-03-25T17:50:22.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.626+0000 7fc24d7fa640 1 -- 172.21.15.44:0/3947268221 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7fc244052140 con 0x7fc25006c380 2024-03-25T17:50:22.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.626+0000 7fc24d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:22.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.626+0000 7fc24d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:50:22.627 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:50:22.627 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:50:22.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 10 librados: watch_flush enter 2024-03-25T17:50:22.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 10 librados: watch_flush exit 2024-03-25T17:50:22.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 20 client.4721.objecter shutdown clearing up homeless session... 2024-03-25T17:50:22.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 10 client.4721.objecter successfully canceled tick 2024-03-25T17:50:22.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 >> v1:172.21.15.44:6800/3257043132 conn(0x7fc23406e0f0 legacy=0x7fc2340705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:22.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 10 monclient: shutdown 2024-03-25T17:50:22.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:50:22.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 >> v1:172.21.15.44:6790/0 conn(0x7fc25006c380 legacy=0x7fc2501c0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:22.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 shutdown_connections 2024-03-25T17:50:22.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 >> 172.21.15.44:0/3947268221 conn(0x7fc2500b5d10 msgr2=0x7fc2500ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:50:22.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 shutdown_connections 2024-03-25T17:50:22.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 1 -- 172.21.15.44:0/3947268221 wait complete. 2024-03-25T17:50:22.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:22.628+0000 7fc257250640 1 librados: shutdown 2024-03-25T17:50:22.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:50:29.449 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.563+0000 7f7fd9d7a640 10 monclient: get_monmap_and_config 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.563+0000 7f7fd9d7a640 10 monclient: build_initial_monmap 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.563+0000 7f7fd9d7a640 10 monclient: monmap: 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:50:29.564550+0000 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:50:29.564550+0000 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:29.564 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:50:29.565 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T17:50:29.565 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T17:50:29.565 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.564+0000 7f7fd9d7a640 1 Processor -- start 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.564+0000 7f7fd9d7a640 1 -- start start 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.564+0000 7f7fd9d7a640 10 monclient: init 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 10 monclient: _reopen_session rank -1 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 10 monclient(hunting): picked mon.noname-b con 0x7f7fd40a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 10 monclient(hunting): picked mon.noname-a con 0x7f7fd413f710 addr v1:172.21.15.44:6789/0 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 10 monclient(hunting): picked mon.noname-c con 0x7f7fd41430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:50:29.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd4142520 con 0x7f7fd413f710 2024-03-25T17:50:29.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40b28a0 con 0x7f7fd41430c0 2024-03-25T17:50:29.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40b2a30 con 0x7f7fd40a98f0 2024-03-25T17:50:29.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 10 monclient(hunting): _renew_subs 2024-03-25T17:50:29.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.565+0000 7f7fd9d7a640 10 monclient(hunting): authenticate will time out at 6632.793413s 2024-03-25T17:50:29.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.580+0000 7f7fd2ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f7fd413f710 0x7f7fd41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57912/0 (socket says 172.21.15.44:57912) 2024-03-25T17:50:29.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.580+0000 7f7fd2ffd640 1 -- 172.21.15.44:0/2686728277 learned_addr learned my addr 172.21.15.44:0/2686728277 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:50:29.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.600+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811289965 0 0) 0x7f7fd40b28a0 con 0x7f7fd41430c0 2024-03-25T17:50:29.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.600+0000 7f7fd27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:50:29.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.600+0000 7f7fd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:29.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.600+0000 7f7fd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:29.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.600+0000 7f7fd27fc640 10 monclient(hunting): my global_id is 4720 2024-03-25T17:50:29.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.600+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fbc003520 con 0x7f7fd41430c0 2024-03-25T17:50:29.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.602+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964803717 0 0) 0x7f7fd4142520 con 0x7f7fd413f710 2024-03-25T17:50:29.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.602+0000 7f7fd27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:50:29.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.602+0000 7f7fd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:29.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.602+0000 7f7fd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:29.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.602+0000 7f7fd27fc640 10 monclient(hunting): my global_id is 4728 2024-03-25T17:50:29.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.602+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd40b28a0 con 0x7f7fd413f710 2024-03-25T17:50:29.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.609+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354525689 0 0) 0x7f7fd40b2a30 con 0x7f7fd40a98f0 2024-03-25T17:50:29.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.609+0000 7f7fd27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:50:29.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.609+0000 7f7fd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:29.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.609+0000 7f7fd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:29.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.609+0000 7f7fd27fc640 10 monclient(hunting): my global_id is 4724 2024-03-25T17:50:29.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.609+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd4142520 con 0x7f7fd40a98f0 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1482286767 0 0) 0x7f7fbc003520 con 0x7f7fd41430c0 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 >> v1:172.21.15.44:6790/0 conn(0x7f7fd40a98f0 legacy=0x7f7fd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 >> v1:172.21.15.44:6789/0 conn(0x7f7fd413f710 legacy=0x7f7fd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 10 monclient: _finish_hunting 0 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 1 monclient: found mon.noname-c 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd40b44c0 con 0x7f7fd41430c0 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 10 monclient: _finish_auth 0 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 10 monclient: _check_auth_tickets 2024-03-25T17:50:29.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fcc001a10 con 0x7f7fd41430c0 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd27fc640 10 monclient: dump: 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd9d7a640 5 monclient: authenticate success, global_id 4720 2024-03-25T17:50:29.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.611+0000 7f7fd9d7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:50:29.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.627+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fcc001b70 con 0x7f7fd41430c0 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.627+0000 7f7fd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.627+0000 7f7fd27fc640 1 -- 172.21.15.44:0/2686728277 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fcc001f80 con 0x7f7fd41430c0 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.627+0000 7f7fd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.627+0000 7f7fd27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.627+0000 7f7fd27fc640 10 monclient: dump: 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:29.628 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:29.629 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:29.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.627+0000 7f7fd9d7a640 10 monclient: get_monmap_and_config success 2024-03-25T17:50:29.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.627+0000 7f7fd9d7a640 4 set_mon_vals no callback set 2024-03-25T17:50:29.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.628+0000 7f7fd9d7a640 10 monclient: shutdown 2024-03-25T17:50:29.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.628+0000 7f7fd9d7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:50:29.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.628+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/2686728277 >> v1:172.21.15.111:6789/0 conn(0x7f7fd41430c0 legacy=0x7f7fd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:29.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.628+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/2686728277 shutdown_connections 2024-03-25T17:50:29.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.628+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/2686728277 >> 172.21.15.44:0/2686728277 conn(0x7f7fd40b7ea0 msgr2=0x7f7fd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:50:29.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.628+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/2686728277 shutdown_connections 2024-03-25T17:50:29.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.628+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/2686728277 wait complete. 2024-03-25T17:50:29.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.629+0000 7f7fd9d7a640 10 monclient: build_initial_monmap 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.629+0000 7f7fd9d7a640 10 monclient: monmap: 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.630+0000 7f7fd9d7a640 1 librados: starting msgr at 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.630+0000 7f7fd9d7a640 1 librados: starting objecter 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.630+0000 7f7fd9d7a640 1 Processor -- start 2024-03-25T17:50:29.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.630+0000 7f7fd9d7a640 1 -- start start 2024-03-25T17:50:29.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.630+0000 7f7fd9d7a640 1 librados: setting wanted keys 2024-03-25T17:50:29.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.630+0000 7f7fd9d7a640 1 librados: calling monclient init 2024-03-25T17:50:29.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.630+0000 7f7fd9d7a640 10 monclient: init 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 10 monclient: _reopen_session rank -1 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 10 monclient(hunting): picked mon.noname-a con 0x7f7fd40a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 10 monclient(hunting): picked mon.noname-c con 0x7f7fd413f710 addr v1:172.21.15.44:6790/0 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 10 monclient(hunting): picked mon.noname-b con 0x7f7fd41430c0 addr v1:172.21.15.111:6789/0 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd4136ec0 con 0x7f7fd40a98f0 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd4136c20 con 0x7f7fd41430c0 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd4137db0 con 0x7f7fd413f710 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 10 monclient(hunting): _renew_subs 2024-03-25T17:50:29.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.631+0000 7f7fd9d7a640 10 monclient(hunting): authenticate will time out at 6632.859279s 2024-03-25T17:50:29.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.634+0000 7f7fd3fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7fd41430c0 0x7f7fd41e2c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:41540/0 (socket says 172.21.15.44:41540) 2024-03-25T17:50:29.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.634+0000 7f7fd3fff640 1 -- 172.21.15.44:0/126726549 learned_addr learned my addr 172.21.15.44:0/126726549 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T17:50:29.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.634+0000 7f7fbbfff640 10 client.?.objecter ms_handle_connect 0x7f7fd41430c0 2024-03-25T17:50:29.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.634+0000 7f7fbbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:29.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.642+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549256342 0 0) 0x7f7fd4136c20 con 0x7f7fd41430c0 2024-03-25T17:50:29.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.642+0000 7f7fbbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:50:29.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.642+0000 7f7fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:29.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.642+0000 7f7fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:29.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.642+0000 7f7fbbfff640 10 monclient(hunting): my global_id is 4723 2024-03-25T17:50:29.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.642+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc0004eb0 con 0x7f7fd41430c0 2024-03-25T17:50:29.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.643+0000 7f7fbbfff640 10 client.?.objecter ms_handle_connect 0x7f7fd40a98f0 2024-03-25T17:50:29.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.643+0000 7f7fbbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.650+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3237633681 0 0) 0x7f7fc0004eb0 con 0x7f7fd41430c0 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.650+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fd4136c20 con 0x7f7fd41430c0 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.650+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fcc001f40 con 0x7f7fd41430c0 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.650+0000 7f7fbbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.650+0000 7f7fbbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.650+0000 7f7fbbfff640 10 monclient(hunting): dump: 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:29.651 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:29.652 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:29.652 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:29.652 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:29.652 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:29.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.650+0000 7f7fbbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:50:29.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.659+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138725447 0 0) 0x7f7fd4136ec0 con 0x7f7fd40a98f0 2024-03-25T17:50:29.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.659+0000 7f7fbbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:50:29.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.659+0000 7f7fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:50:29.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.659+0000 7f7fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:50:29.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.659+0000 7f7fbbfff640 10 monclient(hunting): my global_id is 4731 2024-03-25T17:50:29.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.659+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc0004eb0 con 0x7f7fd40a98f0 2024-03-25T17:50:29.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2355362504 0 0) 0x7f7fd4136c20 con 0x7f7fd41430c0 2024-03-25T17:50:29.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 >> v1:172.21.15.44:6790/0 conn(0x7f7fd413f710 legacy=0x7f7fd40b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:29.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 >> v1:172.21.15.44:6789/0 conn(0x7f7fd40a98f0 legacy=0x7f7fd41376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:29.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 10 monclient: _finish_hunting 0 2024-03-25T17:50:29.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 1 monclient: found mon.b 2024-03-25T17:50:29.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:50:29.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd4138b50 con 0x7f7fd41430c0 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 10 monclient: _finish_auth 0 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 10 monclient: _check_auth_tickets 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 5 monclient: authenticate success, global_id 4723 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 10 monclient: _renew_subs 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd4139830 con 0x7f7fd41430c0 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 10 monclient: _renew_subs 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd41369b0 con 0x7f7fd41430c0 2024-03-25T17:50:29.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.661+0000 7f7fd9d7a640 1 librados: init done 2024-03-25T17:50:29.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.662+0000 7f7fd9d7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:29.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.662+0000 7f7fd9d7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:29.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.662+0000 7f7fd9d7a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:29.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.662+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd4139a20 con 0x7f7fd41430c0 2024-03-25T17:50:29.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.933+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fcc001510 con 0x7f7fd41430c0 2024-03-25T17:50:29.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.933+0000 7f7fbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:50:29.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.933+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1008163629 0 0) 0x7f7fcc003c20 con 0x7f7fd41430c0 2024-03-25T17:50:29.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.933+0000 7f7fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:50:29.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.933+0000 7f7fbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:50:29.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.933+0000 7f7fbbfff640 10 monclient: dump: 2024-03-25T17:50:29.936 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T17:50:29.936 INFO:teuthology.misc.health.smithi044.stderr:fsid 6a379495-4f08-496c-b095-a059f345ad04 2024-03-25T17:50:29.936 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:29.936 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T17:38:09.749652+0000 2024-03-25T17:50:29.936 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T17:50:29.936 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T17:50:29.936 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T17:50:29.937 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T17:50:29.937 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T17:50:29.937 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T17:50:29.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.933+0000 7f7fd1ffb640 4 set_mon_vals no callback set 2024-03-25T17:50:29.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.934+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1628563533 0 0) 0x7f7fcc003e40 con 0x7f7fd41430c0 2024-03-25T17:50:29.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.935+0000 7f7fd37fe640 10 monclient: get_auth_request con 0x7f7fc006dcc0 auth_method 0 2024-03-25T17:50:29.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.935+0000 7f7fbbfff640 10 client.4723.objecter ms_handle_connect 0x7f7fc006dcc0 2024-03-25T17:50:29.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.939+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 274463372 0 0) 0x7f7fcc054a30 con 0x7f7fd41430c0 2024-03-25T17:50:29.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.939+0000 7f7fbbfff640 10 client.4723.objecter ms_dispatch 0x7f7fd4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:50:29.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.939+0000 7f7fbbfff640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:50:29.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.939+0000 7f7fbbfff640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:50:29.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.940+0000 7f7fbbfff640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T17:50:29.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.940+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fcc050540 con 0x7f7fd41430c0 2024-03-25T17:50:29.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.941+0000 7f7fbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:50:29.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:29.941+0000 7f7fbbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:50:30.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.206+0000 7f7fd9d7a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:30.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.206+0000 7f7fd9d7a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:30.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.206+0000 7f7fd9d7a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T17:50:30.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.206+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fd40627b0 con 0x7f7fd41430c0 2024-03-25T17:50:30.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.226+0000 7f7fbbfff640 1 -- 172.21.15.44:0/126726549 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 13350174) 0x7f7fcc052470 con 0x7f7fd41430c0 2024-03-25T17:50:30.226 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.226+0000 7f7fbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:50:30.227 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.226+0000 7f7fbbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:50:30.227 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T17:50:30.227 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"6a379495-4f08-496c-b095-a059f345ad04","created":"2024-03-25T17:39:00.081511+0000","modified":"2024-03-25T17:49:04.142837+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:49:04.142837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"620cf0f3-0d96-4974-984e-50066cd7e73d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e8b5557d-5398-45c4-8d27-c70dce8bdbea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d8e59de1-0a9f-45f7-85f4-1e2f1c6222c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ed0322fe-4a1a-4b8c-9886-9d7b9a18530e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"af25a07f-955a-4cb4-b3b3-cdc2fbfad425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7d1a4285-ebec-44ca-8893-ce7dfe83489c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c5e74c5-4698-4e3f-91c8-5e5134445c37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b36f57af-8e3c-45a4-a735-476c838df28e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:50:30.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 10 librados: watch_flush enter 2024-03-25T17:50:30.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 10 librados: watch_flush exit 2024-03-25T17:50:30.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T17:50:30.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 10 client.4723.objecter successfully canceled tick 2024-03-25T17:50:30.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 >> v1:172.21.15.44:6800/3257043132 conn(0x7f7fc006dcc0 legacy=0x7f7fc0070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:30.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 10 monclient: shutdown 2024-03-25T17:50:30.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:50:30.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 >> v1:172.21.15.111:6789/0 conn(0x7f7fd41430c0 legacy=0x7f7fd41e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:50:30.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 shutdown_connections 2024-03-25T17:50:30.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 >> 172.21.15.44:0/126726549 conn(0x7f7fd40b7ea0 msgr2=0x7f7fd413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:50:30.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 shutdown_connections 2024-03-25T17:50:30.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.228+0000 7f7fd9d7a640 1 -- 172.21.15.44:0/126726549 wait complete. 2024-03-25T17:50:30.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T17:50:30.229+0000 7f7fd9d7a640 1 librados: shutdown 2024-03-25T17:50:30.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:50:30.240 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T17:50:30.242 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T17:50:30.242 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T17:50:30.242 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:50:32.148 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:11:26.607 35502== 2024-03-25T17:50:32.149 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:11:26.607 35502== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:50:32.185 INFO:tasks.ceph.osd.0.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:50:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:50:36.344 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T17:50:36.345 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T17:50:36.345 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T17:50:36.345 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:50:38.419 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:11:32.861 35505== 2024-03-25T17:50:38.419 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:11:32.861 35505== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:50:38.455 INFO:tasks.ceph.osd.1.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:50:39.591 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:56.947 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:39.596 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:56.952 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:39.630 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:56.986 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:39.642 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:56.962 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:39.650 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:57.005 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:39.652 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:56.973 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:39.660 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:57.015 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:39.675 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:56.995 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:39.685 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:57.041 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:50:42.447 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T17:50:42.447 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T17:50:42.447 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T17:50:42.448 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:50:44.410 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:11:38.859 35504== 2024-03-25T17:50:44.410 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:11:38.859 35504== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:50:44.446 INFO:tasks.ceph.osd.2.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:50:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:50:48.550 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T17:50:48.550 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T17:50:48.550 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T17:50:48.551 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:50:50.327 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:11:44.761 35506== 2024-03-25T17:50:50.327 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:11:44.761 35506== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:50:50.367 INFO:tasks.ceph.osd.3.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:50:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:50:54.653 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T17:50:54.653 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T17:50:54.653 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T17:50:54.653 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:50:56.738 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:11:51.187 35158== 2024-03-25T17:50:56.738 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:11:51.187 35158== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:50:56.778 INFO:tasks.ceph.osd.4.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:50:59.627 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:12:16.947 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:59.629 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:16.985 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:59.630 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:12:16.950 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:59.638 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:16.994 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:59.656 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:12:16.976 34603== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:59.662 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:17.018 35186== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:59.683 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:12:17.039 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:59.686 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:12:17.042 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:50:59.710 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:12:17.066 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:51:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:51:00.755 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T17:51:00.755 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T17:51:00.755 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T17:51:00.756 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:51:02.743 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:11:57.188 35159== 2024-03-25T17:51:02.743 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:11:57.188 35159== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:51:02.783 INFO:tasks.ceph.osd.5.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:51:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:51:06.858 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T17:51:06.858 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T17:51:06.858 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T17:51:06.859 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:51:08.717 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:12:03.147 35160== 2024-03-25T17:51:08.717 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:12:03.147 35160== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:51:08.756 INFO:tasks.ceph.osd.6.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:51:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:51:12.960 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T17:51:12.961 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T17:51:12.961 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T17:51:12.961 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:51:14.507 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:12:08.931 35161== 2024-03-25T17:51:14.507 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:12:08.931 35161== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:51:14.545 INFO:tasks.ceph.osd.7.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:51:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:51:19.063 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T17:51:19.064 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T17:51:19.064 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T17:51:19.064 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T17:51:19.064 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:51:19.065 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T17:51:19.065+0000 7f9cc7400640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35198) UID: 0 2024-03-25T17:51:19.092 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T17:51:19.093 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T17:51:19.093 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:51:19.121 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T17:51:19.121 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T17:51:19.121 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T17:51:19.121 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:51:19.176 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-03-25T17:51:19.165+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35158) UID: 0 2024-03-25T17:51:19.176 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-03-25T17:51:19.166+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T17:51:19.638 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:36.994 35186== 2024-03-25T17:51:19.639 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:36.994 35186== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:51:19.648 INFO:tasks.ceph.mon.a.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:51:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:51:25.223 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T17:51:25.223 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T17:51:25.224 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T17:51:25.224 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:51:25.240 INFO:tasks.ceph.mon.c.smithi044.stderr:2024-03-25T17:51:25.228+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35160) UID: 0 2024-03-25T17:51:25.240 INFO:tasks.ceph.mon.c.smithi044.stderr:2024-03-25T17:51:25.230+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T17:51:25.762 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:12:43.118 35192== 2024-03-25T17:51:25.762 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:12:43.118 35192== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:51:25.771 INFO:tasks.ceph.mon.c.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:51:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:51:31.326 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T17:51:31.326 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T17:51:31.327 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T17:51:31.327 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:51:31.342 INFO:tasks.ceph.mon.b.smithi111.stderr:2024-03-25T17:51:31.331+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34599) UID: 0 2024-03-25T17:51:31.343 INFO:tasks.ceph.mon.b.smithi111.stderr:2024-03-25T17:51:31.332+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T17:51:31.889 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:12:49.209 34603== 2024-03-25T17:51:31.906 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:12:49.209 34603== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:51:31.907 INFO:tasks.ceph.mon.b.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:51:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:51:37.429 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T17:51:37.429 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T17:51:37.429 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T17:51:37.430 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T17:51:37.476 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:51:37.476 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T17:51:37.588 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:51:37.588 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T17:51:37.647 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:51:37.647 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T17:51:37.705 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:51:37.705 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T17:51:37.761 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:51:37.762 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T17:51:37.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:51:37.863 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T17:51:37.920 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:51:37.920 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T17:51:37.969 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:51:37.969 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T17:51:38.020 INFO:tasks.ceph:Archiving mon data... 2024-03-25T17:51:38.021 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/data/mon.a.tgz 2024-03-25T17:51:38.021 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-03-25T17:51:38.037 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.GE69Szx4Bb 2024-03-25T17:51:38.037 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GE69Szx4Bb 2024-03-25T17:51:38.136 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.GE69Szx4Bb 2024-03-25T17:51:38.220 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.GE69Szx4Bb is 134KB 2024-03-25T17:51:38.272 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.GE69Szx4Bb 2024-03-25T17:51:38.286 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/data/mon.c.tgz 2024-03-25T17:51:38.287 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-03-25T17:51:38.340 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.ALj9tHSBpk 2024-03-25T17:51:38.340 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ALj9tHSBpk 2024-03-25T17:51:38.428 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.ALj9tHSBpk 2024-03-25T17:51:38.512 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.ALj9tHSBpk is 134KB 2024-03-25T17:51:38.564 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.ALj9tHSBpk 2024-03-25T17:51:38.578 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/data/mon.b.tgz 2024-03-25T17:51:38.578 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-03-25T17:51:38.596 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.VbhGv9fhh3 2024-03-25T17:51:38.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VbhGv9fhh3 2024-03-25T17:51:38.751 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.VbhGv9fhh3 2024-03-25T17:51:38.839 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.VbhGv9fhh3 is 134KB 2024-03-25T17:51:38.891 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.VbhGv9fhh3 2024-03-25T17:51:38.906 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T17:51:38.906 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T17:51:38.909 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T17:51:39.235 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:51:39.476 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T17:51:39.478 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/remote/smithi044/crash 2024-03-25T17:51:39.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T17:51:39.506 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/remote/smithi111/crash 2024-03-25T17:51:39.507 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T17:51:39.532 INFO:tasks.ceph:Compressing logs... 2024-03-25T17:51:39.532 DEBUG:teuthology.orchestra.run.smithi044:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T17:51:39.550 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T17:51:39.574 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T17:51:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T17:51:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T17:51:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34847.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35059.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34847.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34801.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34847.log.gz 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.audit.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35059.log.gz 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T17:51:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34887.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34801.log: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34887.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34801.log.gz 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34928.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34887.log.gz 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35012.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34756.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35012.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34928.log.gz 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34968.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35012.log.gz 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mgr.x.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.34756.log.gz 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.34968.log: -- /var/log/ceph/ceph-mon.b.log 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34968.log.gz 2024-03-25T17:51:39.617 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T17:51:39.618 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T17:51:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40405.log 2024-03-25T17:51:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40914.log 2024-03-25T17:51:39.624 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35749.log 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39976.log 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gz 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.35749.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40914.log.gz 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37580.log 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39976.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35749.log.gz 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40522.log 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.37580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40797.log 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39976.log.gz 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37580.log.gz 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40797.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40522.log.gz 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39742.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.40797.log.gz 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T17:51:39.625 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41382.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41928.log 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-03-25T17:51:39.626 INFO:teuthology.orchestra.run.smithi044.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36628.log 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41031.log 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41928.log.gz 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.36628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39703.log 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36628.log.gz 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41031.log.gz 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40992.log 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.35200.log: /var/log/ceph/ceph-client.admin.39703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40992.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39703.log.gz 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39469.log 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39664.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40992.log.gz 2024-03-25T17:51:39.627 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39820.log 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41538.log 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41460.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39469.log.gz 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41811.log 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39820.log.gz 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41538.log.gz 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41148.log 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41811.log.gz 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40639.log 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-03-25T17:51:39.628 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41148.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40015.log 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42201.log 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41148.log.gz 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40719.log 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40639.log.gz 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42123.log 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40015.log.gz 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40758.log 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42201.log.gz 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42123.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40719.log.gz 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5%/var/log/ceph/ceph-client.admin.40758.log: -- replaced with /var/log/ceph/ceph-client.admin.42123.log.gz 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42362.log 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-03-25T17:51:39.629 INFO:teuthology.orchestra.run.smithi044.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40758.log.gz 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42084.log 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39859.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41577.log 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42362.log.gz 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41577.log: 86.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42084.log.gz 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41577.log.gz 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42162.log 2024-03-25T17:51:39.630 INFO:teuthology.orchestra.run.smithi044.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40171.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41070.log 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42162.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41616.log 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42162.log.gz 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40483.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41070.log.gz 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39898.log 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2024-03-25T17:51:39.631 INFO:teuthology.orchestra.run.smithi044.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41616.log.gz 2024-03-25T17:51:39.632 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40444.log 2024-03-25T17:51:39.632 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42045.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39898.log.gz 2024-03-25T17:51:39.632 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41967.log 2024-03-25T17:51:39.632 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35503.log 2024-03-25T17:51:39.632 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40444.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2024-03-25T17:51:39.632 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2024-03-25T17:51:39.632 INFO:teuthology.orchestra.run.smithi044.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40444.log.gz 2024-03-25T17:51:39.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.35503.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41967.log.gz 2024-03-25T17:51:39.633 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42283.log 2024-03-25T17:51:39.633 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-03-25T17:51:39.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40093.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35503.log.gz 2024-03-25T17:51:39.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39274.log 2024-03-25T17:51:39.633 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39235.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz 2024-03-25T17:51:39.633 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39352.log 2024-03-25T17:51:39.634 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42283.log.gz 2024-03-25T17:51:39.634 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39274.log: gzip -5 --verbose -- 86.9% /var/log/ceph/ceph-client.admin.40875.log 2024-03-25T17:51:39.634 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-03-25T17:51:39.634 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40210.log 2024-03-25T17:51:39.634 INFO:teuthology.orchestra.run.smithi044.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39274.log.gz 2024-03-25T17:51:39.634 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40679.log 2024-03-25T17:51:39.634 INFO:teuthology.orchestra.run.smithi044.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39352.log.gz 2024-03-25T17:51:39.634 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41226.log 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40875.log.gz 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40679.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40210.log.gz 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39430.log 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35604.log 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40679.log.gz 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41226.log.gz 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5/var/log/ceph/ceph-client.admin.35604.log: --verbose -- /var/log/ceph/ceph-client.admin.42323.log 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39430.log.gz 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41499.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29333.log 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42323.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35604.log.gz 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39508.log 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph.tmp-client.admin.29333.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42323.log.gz 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-03-25T17:51:39.635 INFO:teuthology.orchestra.run.smithi044.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29333.log.gz 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39508.log: /var/log/ceph/ceph-client.admin.41850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39195.log 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.3.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39508.log.gz 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41889.log 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42006.log 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41889.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39195.log.gz 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40561.log 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42006.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41889.log.gz 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39625.log 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40561.log: gzip -5 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42006.log.gz 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39547.log 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39625.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40561.log.gz 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T17:51:39.636 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40600.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39547.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39625.log.gz 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41187.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39547.log.gz 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40249.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41187.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40600.log.gz 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.41343.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40249.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41187.log.gz 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41343.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41109.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40249.log.gz 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42240.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41343.log.gz 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38864.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41109.log: /var/log/ceph/ceph-client.admin.42240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41733.log 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.38864.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41109.log.gz 2024-03-25T17:51:39.637 INFO:teuthology.orchestra.run.smithi044.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42240.log.gz 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38864.log.gz 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41733.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40836.log 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.2.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41733.log.gz 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40836.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40836.log.gz 2024-03-25T17:51:39.638 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39781.log 2024-03-25T17:51:39.639 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41772.log 2024-03-25T17:51:39.639 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39313.log: /var/log/ceph/ceph-client.admin.39781.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-03-25T17:51:39.639 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39781.log.gz 2024-03-25T17:51:39.639 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39586.log 2024-03-25T17:51:39.639 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41772.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41772.log.gz 2024-03-25T17:51:39.639 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39937.log 2024-03-25T17:51:39.639 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39937.log: /var/log/ceph/ceph-client.admin.39586.log: 86.6% 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39586.log.gz 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39937.log.gz 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42402.log 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40132.log: /var/log/ceph/ceph-client.admin.41655.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42402.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42402.log.gz 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41421.log 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40366.log 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41421.log: /var/log/ceph/ceph-client.admin.40327.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41421.log.gz 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:51:39.640 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40288.log 2024-03-25T17:51:39.641 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40953.log 2024-03-25T17:51:39.641 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40366.log: /var/log/ceph/ceph-client.admin.40288.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40366.log.gz 2024-03-25T17:51:39.641 INFO:teuthology.orchestra.run.smithi044.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40288.log.gz 2024-03-25T17:51:39.641 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40054.log 2024-03-25T17:51:39.641 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40953.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40953.log.gz 2024-03-25T17:51:39.641 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T17:51:39.641 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40054.log: /var/log/ceph/ceph-mon.c.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40054.log.gz 2024-03-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi044.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T17:51:40.451 INFO:teuthology.orchestra.run.smithi044.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T17:51:40.454 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T17:51:40.475 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T17:51:40.508 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T17:51:40.707 INFO:teuthology.orchestra.run.smithi044.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T17:51:40.707 INFO:teuthology.orchestra.run.smithi044.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T17:51:40.790 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T17:51:40.823 INFO:teuthology.orchestra.run.smithi044.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T17:51:44.665 INFO:teuthology.orchestra.run.smithi044.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T17:51:44.732 INFO:teuthology.orchestra.run.smithi044.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T17:51:44.737 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T17:51:44.737 INFO:teuthology.orchestra.run.smithi044.stderr:real 0m5.172s 2024-03-25T17:51:44.737 INFO:teuthology.orchestra.run.smithi044.stderr:user 0m14.166s 2024-03-25T17:51:44.737 INFO:teuthology.orchestra.run.smithi044.stderr:sys 0m0.527s 2024-03-25T17:51:44.855 INFO:teuthology.orchestra.run.smithi111.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T17:51:44.857 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T17:51:44.857 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m5.269s 2024-03-25T17:51:44.857 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m8.757s 2024-03-25T17:51:44.857 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m0.344s 2024-03-25T17:51:44.857 INFO:tasks.ceph:Archiving logs... 2024-03-25T17:51:44.857 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/remote/smithi044/log 2024-03-25T17:51:44.858 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T17:51:47.039 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/remote/smithi111/log 2024-03-25T17:51:47.040 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T17:51:48.518 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:51:48.716 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e92e78aa9f7345df9d6738de32f64956 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:51:48.720 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T17:51:48.730 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T17:51:48.739 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T17:51:48.739 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T17:51:48.752 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T17:51:48.829 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T17:51:48.837 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T17:51:48.838 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:51:48.840 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-03-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 +526us[ +526us] +/- 82ms 2024-03-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -1226us[-1257us] +/- 51ms 2024-03-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +1081us[+1081us] +/- 71ms 2024-03-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-03-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +588us[ +588us] +/- 82ms 2024-03-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -1220us[-1249us] +/- 51ms 2024-03-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +1141us[+1141us] +/- 71ms 2024-03-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:51:48.861 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T17:51:48.872 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T17:51:48.872 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T17:51:48.882 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T17:51:48.973 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T17:51:49.042 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:51:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:49.074 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:51:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:49.105 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T17:51:49.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T17:51:49.144 INFO:teuthology.task.internal:Duration was 6375.742056 seconds 2024-03-25T17:51:49.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T17:51:49.154 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T17:51:49.154 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T17:51:49.156 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:51:49.446 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T17:51:49.446 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T17:51:49.446 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T17:51:49.467 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T17:51:49.467 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T17:51:49.507 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T17:51:49.507 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T17:51:49.509 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T17:51:49.576 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T17:51:49.576 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T17:51:49.581 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T17:51:49.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T17:51:49.730 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T17:51:49.730 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T17:51:49.757 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T17:51:49.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T17:51:49.794 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:51:49.800 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:51:49.823 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2024-03-25T17:51:49.850 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-03-25T17:51:49.870 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:51:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:49.904 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:51:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:49.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T17:51:49.936 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T17:51:49.937 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/remote/smithi044 2024-03-25T17:51:49.937 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T17:51:49.979 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238/remote/smithi111 2024-03-25T17:51:49.979 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T17:51:50.013 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T17:51:50.013 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T17:51:50.016 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T17:51:50.063 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T17:51:50.074 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T17:51:50.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T17:51:50.087 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T17:51:50.088 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T17:51:50.090 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T17:51:50.102 INFO:teuthology.orchestra.run.smithi044.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 17:51 /home/ubuntu/cephtest 2024-03-25T17:51:50.117 INFO:teuthology.orchestra.run.smithi111.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 17:51 /home/ubuntu/cephtest 2024-03-25T17:51:50.118 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T17:51:50.130 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T17:51:50.161 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T17:51:50.187 DEBUG:teuthology.nuke:shortname: smithi044 2024-03-25T17:51:50.188 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:51:50.210 DEBUG:teuthology.nuke:shortname: smithi111 2024-03-25T17:51:50.210 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:51:50.213 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:56:37.599907', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK5AQGusdjqA96uup2BCfl1loIxJfl6NhrqvGoVH2U8OcUsKi/skdMWIWIDHuAdniI9UIgaBxVWJZrsc7DcoN4s='} 2024-03-25T17:51:50.234 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:56:37.597959', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcLfbwlvh5E0m3Gu/sf3hW7GNawTRIu453B3YyRAHMbkxUusdsYvD1JLkJPyoQpP75IrNRy6bdn/EADUKCRZLQ='} 2024-03-25T17:51:50.265 INFO:teuthology.orchestra.console.smithi044:Power off 2024-03-25T17:51:50.266 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T17:51:50.284 INFO:teuthology.orchestra.console.smithi111:Power off 2024-03-25T17:51:50.284 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T17:51:50.293 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off 2024-03-25T17:51:50.293 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:51:50.306 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-03-25T17:51:50.306 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:51:50.318 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-03-25T17:51:50.329 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-25T17:51:54.318 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:51:54.330 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:51:54.428 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-03-25T17:51:54.444 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-25T17:51:58.433 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:51:58.446 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:51:58.544 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off 2024-03-25T17:51:58.555 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-25T17:51:58.645 INFO:teuthology.orchestra.console.smithi044:Power off completed 2024-03-25T17:52:02.558 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:52:02.668 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-03-25T17:52:02.769 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-03-25T17:52:02.873 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6375.742056131363 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e92e78aa9f7345df9d6738de32f64956 status: fail success: false 2024-03-25T17:52:02.874 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:52:02.963 INFO:teuthology.run:FAIL